-
small script to re-hardlink *ALL* equal files
Have fun, this is not well tested, it will prolly work fine with readonly CD-Images but certainly have problems with hd-installs since there are equal files which *should not* be hardlinked! Take care.
One could probably add filters to the 'find' command to exclude such files. Since I did this as fast hack for CD remastering so i have no need for better selections. I'm working on a much more advanced file-unifier for the vserver project. When that is finished i will announce it here too (ETA: Mid Feburary, i'm in hollyday next time)
Results (compressed image):
691M Jan 2 13:49 KNOPPIX.new
664M Jan 2 17:25 KNOPPIX.new.unified
PS: Klaus ... if you use this for the main Knoppix, the free space you get is dedicated for a recent version of 'distcc', thanks.
Code:
#!/bin/sh
find $1 -type f -exec md5sum {} \; |\
sort +0 |\
{
read sum file;
while [ "$sum" != "" ]; do
while read sum2 file2 && [ $sum = $sum2 ]; do
ln -f "$file" "$file2"
done
sum=$sum2
file="$file2"
done
}
-
Junior Member
registered user
Re: small script to re-hardlink *ALL* equal files
![Quote](images/misc/quote_icon.png)
Originally Posted by
cehteh
Have fun, this is not well tested, it will prolly work fine with readonly CD-Images but certainly have problems with hd-installs since there are equal files which *should not* be hardlinked! Take care.
One could probably add filters to the 'find' command to exclude such files. Since I did this as fast hack for CD remastering so i have no need for better selections. I'm working on a much more advanced file-unifier for the vserver project. When that is finished i will announce it here too (ETA: Mid Feburary, i'm in hollyday next time)
Results (compressed image):
691M Jan 2 13:49 KNOPPIX.new
664M Jan 2 17:25 KNOPPIX.new.unified
PS: Klaus ... if you use this for the main Knoppix, the free space you get is dedicated for a recent version of 'distcc', thanks.
Code:
#!/bin/sh
find $1 -type f -exec md5sum {} \; |\
sort +0 |\
{
read sum file;
while [ "$sum" != "" ]; do
while read sum2 file2 && [ $sum = $sum2 ]; do
ln -f "$file" "$file2"
done
sum=$sum2
file="$file2"
done
}
This looks a little dangerous. Do you realize that this makes all empty (0 bytes) files hardlinks to each other, even locks and logfiles? Also, some files with the same checksum may NOT be actually the same file or have desirably the same contents.
I use a harddisk-installed Debian system that should have all hardlinks intact. If you copied your system from CD, this may not always be the case. You should use rsync -Ha for copying to preserve the hardlinks.
-KK
-
sure .. i know it is a fast hack ... the next tool (vserver unifier) will be much better. Resolving *All* problems u showed.
>This looks a little dangerous. Do you realize that this makes all empty (0 bytes) files hardlinks to each other,
use '! -size 0' within the find
> locks and logfiles
there are no locks when mastering,
logs are not important when kept only in ram ..
! -path '*/var/log*' will fix it anyways
>Also, some files with the same checksum may NOT be actually the same file or have desirably the same contents.
with md5sum the chance of doubletes are microscopic (even less), but you are right .. for the vserver unifier i dont use checksuming
remember this is meant as fast hack and to show a 'general' concept for reducing the diskusage of knoppix.
cheers :)
-
last update
- compares files
- only hardlink files which are not hardlinked yet
- permission and uid/gid must be equal
- size 0 files are excluded
- can be called with multiple arguments which are used directly by find
- writes a script which can be used to undo all hardlinks
i will not do any more updates on this and focus on the vserver unifier now (it will get a small control language and tons of more sophisticated options)
example usage:
./knoppix.unify Path/To/KNOPPIX ! -path "*/var/log/*" >undo_hardlinks.sh
File knoppix.unify:
Code:
#!/bin/bash
find "$@" -type f ! -size 0 -exec md5sum {} \; |\
sort +0 |\
{
read sum file;
while [ "$sum" != "" ]; do
while read sum2 file2 &&\
[ $sum = $sum2 ] &&\
[ $(cut -d " " -f 1 <(ls -i "$file")) != $(cut -d " " -f 1 <(ls -i "$file2")) ] &&\
[ $(cut -c -32 <(ls -ln "$file")) = $(cut -c -32 <(ls -ln "$file2")) ] &&\
cmp "$file" "$file2";
do
ln -f "$file" "$file2" &&
echo "rm -f \"$file2\"; cp -pf \"$file\" \"$file2\""
done
sum=$sum2
file="$file2"
done
}
Similar Threads
-
By fizgig in forum Customising & Remastering
Replies: 1
Last Post: 09-30-2004, 03:35 AM
-
By brazilian in forum General Support
Replies: 2
Last Post: 04-09-2004, 06:10 PM
-
By spydie in forum General Support
Replies: 14
Last Post: 01-31-2004, 07:55 PM
-
By Dave_Bechtel in forum General Support
Replies: 0
Last Post: 06-04-2003, 11:54 AM
-
By .-=Ronin=-. in forum General Support
Replies: 2
Last Post: 04-24-2003, 07:45 PM
Posting Permissions
- You may not post new threads
- You may not post replies
- You may not post attachments
- You may not edit your posts
-
Forum Rules
![Dell PowerEdge R730 2x E5-2699V3 2.3Ghz 36 Core 128GB RAM H730 X520-I350 2x750W picture](/store/img/g/W38AAOSwtcVjVC0Z/s-l225/Dell-PowerEdge-R730-2x-E5-2699V3-2-3Ghz-36-Core-12.jpg)
Dell PowerEdge R730 2x E5-2699V3 2.3Ghz 36 Core 128GB RAM H730 X520-I350 2x750W
$329.99
![Dell PowerEdge R630 Server 2x E5-2680 V4 = 28 Cores S130 32GB RAM NEW 480GB SSD picture](/store/img/g/vx0AAOSwvwFlExVY/s-l225/Dell-PowerEdge-R630-Server-2x-E5-2680-V4-28-Cores-.jpg)
Dell PowerEdge R630 Server 2x E5-2680 V4 = 28 Cores S130 32GB RAM NEW 480GB SSD
$197.99
![Dell Poweredge R630 Server 2x E5-2620 V4 =16 Cores | S130 | 32GB RAM | 2x trays picture](/store/img/g/GAcAAOSw9Q1mGSZW/s-l225/Dell-Poweredge-R630-Server-2x-E5-2620-V4-16-Cores-.jpg)
Dell Poweredge R630 Server 2x E5-2620 V4 =16 Cores | S130 | 32GB RAM | 2x trays
$159.99
![Dell Poweredge R730xd LFF 14-Bay 2U Server | Choose Your CPU & RAM Config picture](/store/img/g/CdkAAOSwY0BmYMoF/s-l225/Dell-Poweredge-R730xd-LFF-14-Bay-2U-Server-Choose-.jpg)
Dell Poweredge R730xd LFF 14-Bay 2U Server | Choose Your CPU & RAM Config
$489.99
![Dell PowerEdge R630 Server 2x E5-2640v3 2.60Ghz 16-Core 64GB H330 picture](/store/img/g/ZD0AAOSwm41mOZYw/s-l225/Dell-PowerEdge-R630-Server-2x-E5-2640v3-2-60Ghz-16.jpg)
Dell PowerEdge R630 Server 2x E5-2640v3 2.60Ghz 16-Core 64GB H330
$182.65
![1U Supermicro Server 10 Bay 2x Intel Xeon 3.3Ghz 8C 128GB RAM 240GB SSD 2x 10GBE picture](/store/img/g/7KoAAOSwephiKTf4/s-l225/1U-Supermicro-Server-10-Bay-2x-Intel-Xeon-3-3Ghz-8.jpg)
1U Supermicro Server 10 Bay 2x Intel Xeon 3.3Ghz 8C 128GB RAM 240GB SSD 2x 10GBE
$259.00
![Dell PowerEdge R720xd 26HDD 300gb 2.5-inch E5-2697 X 2CPU 384RAM 7.2 Tb HDD  picture](/store/img/g/yAwAAOSwdyNlgQ8d/s-l225/Dell-PowerEdge-R720xd-26HDD-300gb-2-5-inch-E5-2697.jpg)
Dell PowerEdge R720xd 26HDD 300gb 2.5-inch E5-2697 X 2CPU 384RAM 7.2 Tb HDD 
$180.00
![Dell PowerEdge T620 8-Bay LFF Xeon E5-2660 0 2.20GHz 48GB NO HDD S110 Server picture](/store/img/g/ZfUAAOSw~TFmclF7/s-l225/Dell-PowerEdge-T620-8-Bay-LFF-Xeon-E5-2660-0-2-20G.jpg)
Dell PowerEdge T620 8-Bay LFF Xeon E5-2660 0 2.20GHz 48GB NO HDD S110 Server
$174.99
![HPE ProLiant DL360 Gen9 4LFF 2x E5-2667v3 3.2GHz =16 Cores 32GB P440AR 4xRJ45 picture](/store/img/g/xFoAAOSwbzdmNL82/s-l225/HPE-ProLiant-DL360-Gen9-4LFF-2x-E5-2667v3-3-2GHz-1.jpg)
HPE ProLiant DL360 Gen9 4LFF 2x E5-2667v3 3.2GHz =16 Cores 32GB P440AR 4xRJ45
$216.00
![1U Server ( 4x VM's ) Hosting VPS Short Depth 20](/store/img/g/mY0AAOSwFDNhGq7d/s-l225/1U-Server-4x-VM-s-Hosting-VPS-Short-Depth-20-1U-Xe.jpg)
1U Server ( 4x VM's ) Hosting VPS Short Depth 20" 1U Xeon 3.5Ghz 16GB X10SLM-F
$149.00