[Shell]Cleanly split large Inbox file

Question

Hello,

Regularly, we have users who let their Inbox file in Thunderbird "inflate" and it then reaches the limit of 4GB!

The solution I have found so far is to manually split the file into several files of 1,000,000 lines (about 740 MB per file) and then combine the end of the first file with the beginning of the second to reconstitute the last email of the split file, the end of the second with the beginning of the third, and so on...

I think it is feasible to script all this in shell.
I already know how to split my file into several parts:

split -1000000 Inbox archive

With that, I get 6 files named "archiveaa", "archiveab", ... "archiveaf".

This part is simple... However, for the adjustment, I’m not quite sure how to proceed.

I know that a message in an Inbox file starts with the pattern "From - ". But how can I tell the script to take the last occurrence of this pattern, cut what follows, and paste it at the beginning of the next file to reconstitute the message? I’m stuck here.

Should I use sed or awk? Does anyone have a lead for the second part?

Thank you in advance!

wolfen · Answer

I answer myself, after two days of research and tinkering...

Here is a somewhat convoluted script but which works under Linux (Ubuntu 10.04 LTS):

#!/bin/sh

# shell script "decoupagembox.sh" version Linux - Ubuntu 10.04 LTS
#
# This script splits a file in mbox format into several archives.
# The number of emails per archive is adjustable. The size of each archive
# will depend on the weight of emails and thus the attachments related.

# February 2011
# code under Creative Commons by-nc-sa license (free distribution and modification
# by citing the original author and without commercial use
#
# author: Damien MAURAN,

echo "Splitting a mbox format file into several archives ..."
echo

if test "$1" = ""
then echo "Please provide the name of the file to process, the number of emails per archive"
echo "and the destination path. Example:"
echo
echo "decoupagembox.sh source_file_name xx destination_path/"
echo
echo "where xx is the number of emails per archive. If xx is not specified,"
echo "the default value will be 500 emails per archive."
echo
echo "End of script ... no processing done."
exit 0
fi

if test "$3" = ""
then echo "Please provide the name of the file to process, the number of emails per archive"
echo "and the destination path. Example:"
echo
echo "decoupage.sh file_name xx destination_path/"
echo
echo "where xx is the number of emails per archive. If xx is not specified,"
echo "the default value will be 500 emails per archive."
echo
echo "End of script ... no processing done."
exit 0
fi

if test "$2" = ""
then X=$((500))
else
X=$(($2))
fi

start_date=$(date)

echo "** Start of processing: $start_date **"
echo
echo "isolation of emails: 1 email = 1 file"
echo
echo "Please wait, this may take some time..."
echo

awk '/From - /{n++}{print >"/tmp/mboxout" n ".txt" }' $1

ls /tmp/mboxout*.txt | wc -l > count.txt
nb='cat count.txt'

rm count.txt

number=$(($nb))
echo "Number of extracted emails: $number"

count1=$(($nb/$X))
echo "Number of packages of $X emails: $count1 packages + 1 package of remaining emails"
echo
echo "Grouping emails to create archives"
echo
echo "Please wait..."
echo

count2=$((count1+1))

for i in $(seq 0 1 $count2)
do
nb_file=$(($i*$X));
end_file=$(($nb_file+$X));
for j in $(seq $nb_file 1 $end_file)
do
if test -f "/tmp/mboxout$j.txt"
then cat /tmp/mboxout$j.txt >> archive$i;
fi
done;
done

rm /tmp/mboxout*.txt

echo "The following archive files have been created:"
ls -x archive*
echo
mv archive* $3
echo "They have been saved in: $3/"
echo

end_date=$(date)
echo "** end of processing: $end_date **"
exit 0

And just in case, here's the same code but optimized for Mac OS X (version 10.6.5):
#!/bin/sh

# shell script "decoupagembox.sh" version Mac OS X (version 10.6.5)
#
# This script splits a file in mbox format into several archives.
# The number of emails per archive is adjustable. The size of each archive
# will depend on the weight of emails and thus the attachments related.

# February 2011
# code under Creative Commons by-nc-sa license (free distribution and modification
# by citing the original author and without commercial use
#
# author: Damien MAURAN,
# dmauran@gmail.com

echo "Splitting a mbox format file into several archives ..."
echo

if test "$1" = ""
then echo "Please provide the name of the file to process, the number of emails per archive"
echo "and the destination path. Example:"
echo
echo "decoupage.sh file_name xx destination_path/"
echo
echo "where xx is the number of emails per archive. If xx is not specified,"
echo "the default value will be 500 emails per archive."
echo
echo "End of script ... no processing done."
exit 0
fi

if test "$3" = ""
then echo "Please provide the name of the file to process, the number of emails per archive"
echo "and the destination path. Example:"
echo
echo "decoupage.sh file_name xx destination_path/"
echo
echo "where xx is the number of emails per archive. If xx is not specified,"
echo "the default value will be 500 emails per archive."
echo
echo "End of script ... no processing done."
exit 0
fi

if test "$2" = ""
then X=$((500))
else
X=$(($2))
fi

start_date=$(date)

echo "** Start of processing: $start_date **"
echo
echo "isolation of emails: 1 email = 1 file"
echo
echo "Please wait, this may take some time..."
echo

awk '/From - /{n++}{filename = "/tmp/mboxout"n".txt"; print >> filename;close (filename) }' $1

ls /tmp/mboxout*.txt | wc -l > count.txt
nb='cat count.txt'

rm count.txt

number=$(($nb))
echo "Number of extracted emails: $number"

count1=$(($nb/$X))
echo "Number of packages of $X emails: $count1 packages + 1 package of remaining emails"
echo
echo "Grouping emails to create archives"
echo
echo "Please wait..."
echo

count2=$((count1+1))

i=$((0))
while [ $i -lt $count2 ]
do
nb_file=$(($i*$X));
end_file=$(($nb_file+$X));
j=$(($nb_file))
while [ $j -lt $end_file ]
do
if test -f "/tmp/mboxout$j.txt"
then cat /tmp/mboxout$j.txt >> archive$i;
fi
true $((j++))
done;
true $((i++))
done

rm /tmp/mboxout*.txt

echo "The following archive files have been created:"
ls -x archive*
echo
mv archive* $3
echo "They have been saved in: $3/"
echo

end_date=$(date)
echo "** end of processing: $end_date **"
exit 0

wolfen13fr · Answer

I answer myself, after two days of research and tinkering...

Here is a somewhat convoluted script but which works under Linux (Ubuntu 10.04 LTS):

#!/bin/sh

# shell script "decoupagembox.sh" version Linux - Ubuntu 10.04 LTS
#
# This script splits a file in mbox format into several archives.
# The number of emails per archive is adjustable. The size of each archive
# will depend on the weight of emails and thus the attachments related.

# February 2011
# code under Creative Commons by-nc-sa license (free distribution and modification
# by citing the original author and without commercial use
#
# author: Damien MAURAN,

echo "Splitting a mbox format file into several archives ..."
echo

if test "$1" = ""
then echo "Please provide the name of the file to process, the number of emails per archive"
echo "and the destination path. Example:"
echo
echo "decoupagembox.sh source_file_name xx destination_path/"
echo
echo "where xx is the number of emails per archive. If xx is not specified,"
echo "the default value will be 500 emails per archive."
echo
echo "End of script ... no processing done."
exit 0
fi

if test "$3" = ""
then echo "Please provide the name of the file to process, the number of emails per archive"
echo "and the destination path. Example:"
echo
echo "decoupage.sh file_name xx destination_path/"
echo
echo "where xx is the number of emails per archive. If xx is not specified,"
echo "the default value will be 500 emails per archive."
echo
echo "End of script ... no processing done."
exit 0
fi

if test "$2" = ""
then X=$((500))
else
X=$(($2))
fi

start_date=$(date)

echo "** Start of processing: $start_date **"
echo
echo "isolation of emails: 1 email = 1 file"
echo
echo "Please wait, this may take some time..."
echo

awk '/From - /{n++}{print >"/tmp/mboxout" n ".txt" }' $1

ls /tmp/mboxout*.txt | wc -l > count.txt
nb='cat count.txt'

rm count.txt

number=$(($nb))
echo "Number of extracted emails: $number"

count1=$(($nb/$X))
echo "Number of packages of $X emails: $count1 packages + 1 package of remaining emails"
echo
echo "Grouping emails to create archives"
echo
echo "Please wait..."
echo

count2=$((count1+1))

for i in $(seq 0 1 $count2)
do
nb_file=$(($i*$X));
end_file=$(($nb_file+$X));
for j in $(seq $nb_file 1 $end_file)
do
if test -f "/tmp/mboxout$j.txt"
then cat /tmp/mboxout$j.txt >> archive$i;
fi
done;
done

rm /tmp/mboxout*.txt

echo "The following archive files have been created:"
ls -x archive*
echo
mv archive* $3
echo "They have been saved in: $3/"
echo

end_date=$(date)
echo "** end of processing: $end_date **"
exit 0

And just in case, here's the same code but optimized for Mac OS X (version 10.6.5):
#!/bin/sh

# shell script "decoupagembox.sh" version Mac OS X (version 10.6.5)
#
# This script splits a file in mbox format into several archives.
# The number of emails per archive is adjustable. The size of each archive
# will depend on the weight of emails and thus the attachments related.

# February 2011
# code under Creative Commons by-nc-sa license (free distribution and modification
# by citing the original author and without commercial use
#
# author: Damien MAURAN,
# dmauran@gmail.com

echo "Splitting a mbox format file into several archives ..."
echo

if test "$1" = ""
then echo "Please provide the name of the file to process, the number of emails per archive"
echo "and the destination path. Example:"
echo
echo "decoupage.sh file_name xx destination_path/"
echo
echo "where xx is the number of emails per archive. If xx is not specified,"
echo "the default value will be 500 emails per archive."
echo
echo "End of script ... no processing done."
exit 0
fi

if test "$3" = ""
then echo "Please provide the name of the file to process, the number of emails per archive"
echo "and the destination path. Example:"
echo
echo "decoupage.sh file_name xx destination_path/"
echo
echo "where xx is the number of emails per archive. If xx is not specified,"
echo "the default value will be 500 emails per archive."
echo
echo "End of script ... no processing done."
exit 0
fi

if test "$2" = ""
then X=$((500))
else
X=$(($2))
fi

start_date=$(date)

echo "** Start of processing: $start_date **"
echo
echo "isolation of emails: 1 email = 1 file"
echo
echo "Please wait, this may take some time..."
echo

awk '/From - /{n++}{filename = "/tmp/mboxout"n".txt"; print >> filename;close (filename) }' $1

ls /tmp/mboxout*.txt | wc -l > count.txt
nb='cat count.txt'

rm count.txt

number=$(($nb))
echo "Number of extracted emails: $number"

count1=$(($nb/$X))
echo "Number of packages of $X emails: $count1 packages + 1 package of remaining emails"
echo
echo "Grouping emails to create archives"
echo
echo "Please wait..."
echo

count2=$((count1+1))

i=$((0))
while [ $i -lt $count2 ]
do
nb_file=$(($i*$X));
end_file=$(($nb_file+$X));
j=$(($nb_file))
while [ $j -lt $end_file ]
do
if test -f "/tmp/mboxout$j.txt"
then cat /tmp/mboxout$j.txt >> archive$i;
fi
true $((j++))
done;
true $((i++))
done

rm /tmp/mboxout*.txt

echo "The following archive files have been created:"
ls -x archive*
echo
mv archive* $3
echo "They have been saved in: $3/"
echo

end_date=$(date)
echo "** end of processing: $end_date **"
exit 0

[Shell]Cleanly split large Inbox file

2 answers

How to plug a usb cord into a tv

Google maps does not recognize my current location [pc]

Doctolib document

Gmx email address blocked

Electronic invoicing

Anomalies in my laptop usage: virus?

I don’t see the wordpress executable.

This solar-powered, gps-enabled agricultural robot sows and weaves weeds by itself (and without pesticides)

Missing lock icon

Avoid driving on august 1st, the worst day for summer traffic.