Puzzle for Programmers

[Blyiss]

Hello, guys. I need a script for moving files.
There are a lot of examples and tutorials but I can't find anything like this:
I have a folder "DIR" that contains subfolders "DIR/a", "DIR/b" and "DIR/c". And each subfolder contains files ",1", ",2" and ",3".
Question: how can I move these files to the subfolder DIR/x in such a manner that files would be renamed and lined up in the following order: "1, 2, 3, 4, 5, 6, 7, 8, 9", if the folder DIR/x is empty. And if it already contains some files, the order would continue starting from the last name of the file like : "10, 11, 12 and so on" ?

Will be very grateful for any help.

[Grifter]

better explanation would be nice

[Blyiss]

better explanation would be nice

OK, I will try. As you know, I am playing with nmh, which is a flavor of mh emails managing program - Message Handling System. When I delete messages in mh, they are not actually deleted but renamed by adding a "," character at the beginning of the file. But mh thinks they are gone and doesn't list them any more. So there is no possibility to view them again in MH, until your rename them back. MH files have very simple names: "1", "2", "3" and so on. When they are deleted, they become ",1", ",2", ",3" and so on. And such a thing happens to any folder like "Inbox", "Outbox","MY Favourite Emails", etc. It is not a big deal to remove these files permanently using a simple one-line bash script. But I want to rename them back so that they become visible again and move to the "Trash" folder, where I can keep them and do to them whatever I want. And here appear two problems:

1. What pattern can be applied for renaming the files with such short names, and
2. How to move the files with the same names from multiple folders to a single one. They will be overwritten, I guess.

I believe there should be a way like first collecting files names somewhere in the memory and then renaming them in order. For example, if I have file ",1" from "Inbox", file ",1" from "Outbox" and file ",1" from "Drafts", they would become "1", "2", "3" and go to the "Trash" folder. But there's another problem. If the "Trash" folder already contains files "1", "2", "3", they will be overwritten. That's why it would be nicer to check this folder first and, if it is not empty, to start renaming files beginning not from "1" but the name or number of the last file like "3" in this case so that the final result for six files would be "1","2","3","4","5","6", containing in the "Trash"
Where can I find info about how to do this?

[Grifter]

can you run

head -n 1 ,1

(this would be on a ,1 file of course)
perhaps a mail can be renamed to a part of its header, that might organize a little, but i'm unsure if they will be able to be read by... well whatever you use to read them, try to copy a ,1 file to tempname, and see if you can view the mail when it's called tempname, or if they absolutely _have_ to be named a digit

[Grifter]

also check if they can have leading zeroes, like be called 00001 instead of 1

[Grifter]

if you can have leading zeroes, it would be rather simple to make a move script, but if you can't it's going to be a little harder due to how ls sorts files ( 1 10 11 2 3 4 5 etc)

assuming you can have leading zeroes, and we decide to set 5 leading zeroes infront of mails (a theoretical max of 99 999 mails before the script falls apart) it would be simple to set a variable like this,

VARCOUNT=`ls /path/to/zeromails | tail -n 1`

now we'd have the number of the latest mail we added, then we simply add 1 to the var, create a loop where we use find to find all the ,x files, and in each iteration we we move the ,x file to /path/to/zeromails renaming it to the VARCOUNT variable, and then add 1 to the variable again, and close the loop, then it'll go for each iteration adding all the ,x mails as one number higher

[Grifter]

of course, if we _can't_ have leading zeroes it's going to take a little more overhead, because when we create the variable we're going to have to put in multiple checks along the way, to first check if there are any filenames with say 5 digits, if not move down to 4 digits, if not move down to 3 digits, if we get a bite here, then we'll have to find the highest 3 digit number which is essentially the same, but as you can see it's a bit more overhead

[Blyiss]

Sorry, I am not very experience in bash scripting. It is interesting, of course. Maybe in the future I will find some time to improve my bash understanding.

I checked. Digits are OK. MH reads any sort of digits in any order including any number of leading zeros. But characters don't work. MH doesn't accept characters.

Code: Select all

: ~/Mail/spam
$ head -n 1 ,1
Return-Path: <jna@ono.com>

What does it mean? Some email address.

[Blyiss]

The guys from Linux forum say that bash cannot cope with such a task. They advise to use Perl instead. I'll take a look what Perl can do.

[drl]

Hi, Blyiss.

Following what I understood your requirements to be, the result of running a small demo script are these lines:

Code: Select all

% ./s1

 Current state:
drafts:
,1

inbox:
,1

outbox:
,01  ,02

trash:


 Final contents of trash:
1  2  3  4

Here is the script that does this:

Code: Select all

#!/bin/sh

# @(#) s1       Demonstrate move and rename.

# Provide audit trail for learning and debugging.
# Interchange the two debug lines to toggle printing on and off.

debug="echo"
debug=":"

# Set up the test situation.
# For production, comment out the remove and create calls.

./remove
./create

echo
DIRS="drafts inbox outbox"
echo " Current state:"
ls -R $DIRS trash

echo
$debug " audit trail of operations:"
$debug

for d in $DIRS
do
  for file in $(ls $d)
  do
    $debug " Working on $d/$file"
    n=${file#,}
    $debug " extracted :$n: from :$file: in :$d:"
    m=$( expr $n + 0 )
    $debug " transformed $n into $m"
    xform=$m
    if [ -f trash/$xform ]
    then
      highest=$( expr $highest + 1 )
      $debug " set highest number to :$highest:"
      bump=$highest
      mv $d/$file trash/$bump
      $debug " moved (bumped) $d/$file to trash/$bump"
    else
      # echo mv $d/$xform trash
      mv $d/$file trash/$xform
      $debug " moved $d/$file to trash/$xform"
      highest=$xform
      $debug " set highest number to :$highest:"
    fi
  done
done

echo
echo " Final contents of trash:"
ls trash

exit 0

An important part of this is "normalizing" the numeric name so that leading zeroes are eliminated. Use as you will; as usual, no warranty ... cheers, drl

[Grifter]

The guys from Linux forum say that bash cannot cope with such a task. They advise to use Perl instead. I'll take a look what Perl can do.

Ech, people only say such a thing if they themselves don't know how to do it

drl - darn you beat me to it, but oh well, here's my version too, and while I don't think it will set fire to your house or impregnate your cat, who can be sure:

Code: Select all

#!/bin/sh

# Set this dir to be the location of where you want
# the files moved, be aware that this should be the
# dump for all deleted mails, it should not be an
# existing mailbox that has to compete with 'real'
# mails
UN=/changeme

# Set this dir to be the root of the mailstructure
# you want to search for ",filenames"
DEUX=/changeme

# Check for the latest mail, stripping the padding zeroes
# (let doesn't preserve the padding zeroes, so we need to
# re-pad them later on)
VARCOUNT=`ls $UN | sed -r 's/(0*)(.*)/\2/' | tail -n 1`

# Check if it's the first run, if not we'll let the var
# iterate higher, so the next filename won't overwrite
# the previous mail
if [ -z $VARCOUNT ]; then
   VARCOUNT=00001
   else
   let VARCOUNT+=1
fi

# Just making sure something isn't funky
if [ -f $VARCOUNT ]; then
        echo "ARRGH! File exists when it shouldn't! Funky Alert!"
        exit 1
fi

# Here we start the loop to find ",filenames", when it finds one
# it will test how many characters the current count has, first
# testing for 5 characters where no padding is needed, if the
# var doesn't have 5 chars it will test for 4, and pad one zero
# infront of the count, otherwise it will continue down the list,
# adding extra zeroes as needed, to make it 5 characters
for i in $( find $DEUX -name ',*'); do
        if [ -n "`echo $VARCOUNT | grep .....`" ]; then mv ${DEUX}/${i} ${UN}/${VARCOUNT}
        elif [ -n "`echo $VARCOUNT | grep ....`" ]; then mv ${DEUX}/${i} ${UN}/0${VARCOUNT}
        elif [ -n "`echo $VARCOUNT | grep ...`" ]; then mv ${DEUX}/${i} ${UN}/00${VARCOUNT}
        elif [ -n "`echo $VARCOUNT | grep ..`" ]; then mv ${DEUX}/${i} ${UN}/000${VARCOUNT}
        elif [ -n "`echo $VARCOUNT | grep .`" ]; then mv ${DEUX}/${i} ${UN}/0000${VARCOUNT}
        fi

# ",file" has been renamed to VARCOUNT with leading zeroes if needed, now we
# increase the count by one to prepare for the next iteration of the loop
let VARCOUNT+=1

# Closing loop
done

[Blyiss]

Thank you, drl, for the script. It works great!
At first it complained that "ls: command not found", so I added:

Code: Select all

PATH=/usr/local/bin:/bin:/sbin:/usr/bin:/usr/sbin
MAILDIR=$HOME/Mail

And it went very well except that it moved to trash all the files it found:
(those "non-numeric arguments" are left-overs from the previous experiments)

Code: Select all

Current state:
drafts:
?1  #1#  ,2

inbox:
,00000000000000000000000001  11  ,14  *17  *19  *20  ,22  *23  26   ,27  3  7
00000000000004               12  ,15  18   2    21   23   ,24  ,26  28   5  ,8
100                          13  ,16  ,19  ,20  *21  ,23  25   27   ,28  6  9

outbox:
1   11  13  15  17  19  20  22  3  5  7  9
10  12  14  16  18  2   21  23  4  6  8

trash:
1

trash:
1

expr: non-numeric argument
expr: non-numeric argument
mv: cannot stat `inbox/100': No such file or directory
expr: non-numeric argument
expr: non-numeric argument
expr: non-numeric argument
expr: non-numeric argument
expr: non-numeric argument
expr: non-numeric argument
mv: `trash/?1' and `trash/?1' are the same file
mv: `trash/11' and `trash/11' are the same file
mv: `trash/21' and `trash/21' are the same file
mv: `trash/31' and `trash/31' are the same file
expr: non-numeric argument
mv: `trash/#1#' and `trash/#1#' are the same file
expr: non-numeric argument
mv: `trash/*17' and `trash/*17' are the same file
expr: non-numeric argument
mv: `trash/*19' and `trash/*19' are the same file
expr: non-numeric argument
mv: `trash/*20' and `trash/*20' are the same file
expr: non-numeric argument
mv: `trash/*21' and `trash/*21' are the same file
expr: non-numeric argument
mv: `trash/*23' and `trash/*23' are the same file

 Final contents of trash:
1  ?1  #1#  10  11  12  13  14  15  16  *17  *19  2  *20  *21  *23  3

Some emails disappeared. The amount of emails in the trash folder and the sum of emails from inbox , outbox, and drafts is not equal.
Here is one more example:

Code: Select all

 Current state:
drafts:

inbox:
,1  ,2

outbox:
1  2  3  4

trash:

trash:


 Final contents of trash:
10  11  12  7  8  9

Its OK here, except that it grabbed all the emails it found.
Can it be because I commented out ./remove and ./create files?
I will tray it a bit later again when I collect enough of emails

[drl]

Hi, Blyiss.

I intended it to be for demonstration. For production, one would add lots of protection.

No, the create/remove commands (scripts) would not cause that. The script assumes that you have only files of the name format "comma followed by decimal numbers".

To cause the extra files to be ignored, you could replace the line:

Code: Select all

for file in $(ls $d)

with

Code: Select all

for file in $( ls -1 $d/ | grep '^,[0-9][0-9]*$' )

Then you would see something like this, noting that I added a few junk file names to draft and outbox:

Code: Select all

% ./s1

 Current state:
drafts:
#1#  ,1  ,44fox  ?1

inbox:
,1

outbox:
,01  ,02  ,hello,world  list,3

trash:


 Final contents of trash:
1  2  3  4

Best wishes ... cheers, drl

[Blyiss]

Than you, Grifter, for a great script!
But if the trash folder is not in the mailbox, I won't be able to see its content from MH. I tried anyway:

Code: Select all

 rex: ~/Mail
$ ./movetotrash2
mv: cannot stat `/home/rex/Mail//home/rex/Mail/spam/,1': No such file or directory
mv: cannot stat `/home/rex/Mail//home/rex/Mail/inbox/,12': No such file or directory
mv: cannot stat `/home/rex/Mail//home/rex/Mail/inbox/,11': No such file or directory
mv: cannot stat `/home/rex/Mail//home/rex/Mail/inbox/,10': No such file or directory

It seems that it tries to count the files that don't exist.

[Grifter]

it looks like it does the path twice, it could be because i got overintentive; in the if lines originally i had the lines be

then mv $i ${UN}/${VARCOUNT}

but right before posting i thought, oh hmm, well maybe i can foolproof it by putting the the $DEUX path before $i, try changing the ${DEUX}/${i} to just $i, in all those if lines, and that should clear it up

[Blyiss]

Thank you, guys, you are great programmers.
Dri, I changed the line to

Code: Select all

for file in $( ls -1 $d/ | grep '^,[0-9][0-9]*$' )

It removes files OK. But some files in the trash were overwritten. The output should be 20 files:

Code: Select all

Current state:
drafts:
1  ,1  15  ,15  16  ,16

inbox:
1  ,1  7  ,7  8  ,8  9  ,9

outbox:
1  ,1  3  ,3  6  ,6

trash:
1  10  15  16  17  3  6  7  8  9


 Final contents of trash:
1  10  15  16  17  2  3  4  5  6  7  8  9

Grifters's version is really great!!!
I changed ${DEUX}/${i} to just $i,
and it works just fine, exactly like a charm. Thanks a lot!
Probably I misunderstood a bit when you said that trash folder should not be an existing mailbox that has to compete with 'real' mails. I made trash right in the folder which MH considers to be a real mailbox (I mean ~/Mail, it reads email from the spoolfolder, "inbox" in my case), and the script works just wonderful! Thank you, very much!!

[Blyiss]

Sorry, Grifter, after posting I notices that trash inside the mailbox doesn't work. But it is fine, when outside ~/Mail folder.

Code: Select all

$ ./movetotrash2
./movetotrash2: line 25: let: 10~: syntax error in expression (error token is "~")
./movetotrash2: line 50: let: 10~: syntax error in expression (error token is "~")
./movetotrash2: line 50: let: 10~: syntax error in expression (error token is "~")
./movetotrash2: line 50: let: 10~: syntax error in expression (error token is "~")

Is there a way to put the files to the trash that is inside the mailbox? Otherwise Mutt doesn't except the folder, if it didn't compiled it by itself. I didn't try Emacs yet.

[Grifter]

Right it should be a real valid mailbox, but it should only contain files that have been dumped there with the script, it shouldn't generate its own mails or move 'real' mails in there, only the dumped stuff should be there, otherwise when it performs VARCOUNT, it will probably do weird stuff

Sorry, Grifter, after posting I notices that trash inside the mailbox doesn't work. But it is fine, when outside ~/Mail folder.

Code: Select all

$ ./movetotrash2
./movetotrash2: line 25: let: 10~: syntax error in expression (error token is "~")

Is there a way to put the files to the trash that is inside the mailbox? Otherwise Mutt doesn't except the folder, if it didn't compiled it by itself. I didn't try Emacs yet.

The error there seems to be the cause of a ~, maybe one of the mails has a ~ after it, when VARCOUNT gets the number, it strips the leading zeroes, but expects only digits in the filename, if a filename is called 00011~ it would probably error the same way you posted, if the reader is the one that puts ~ in the filename that character would have to be stripped away too, which is a pretty simple thing to do

I'm not really sure what you mean by trash files inside the mailbox, I guess it has to do with the the paths

for example, let's say you are in your ~/Mail dir, if you type there, find -name ',*' it would locate all the ,filenames with the relative path, ie it would say ./somedir/,1

but if you instead typed (in that same dir) find /home/luser/mail -name ',*', it would now print the full path

if you set UN=/home/luser/Mail/dumpdir and DEUX=/home/luser/Mail in the script, it should work fine, the beginning of the loop states, find $DEUX -name, so if DEUX has the full path, it would mean find /home/luser/Mail -name, and when i gets its value it should be the filename with the full path

but the errors you got, were line 25 and line 50, the "let" lines, where it's supposed to move the number one step higher, but assuming it thinks the number is 10~, it doesn't know how to change it up to 11, because of the ~ character which I assume exists on one (or more) file(s)

to bypass that, the line that says

VARCOUNT=`ls $UN | sed -r 's/(0*)(.*)/\2/' | tail -n 1`

change that to

VARCOUNT=`ls $UN | sed -r 's/(0*)(.*)/\2/' | tail -n 1 | sed 's/~//g'`

basically just piping it to a sed 's/~//g', to remove this tilde character, the filename with the tilde would remain unchanged but it wouldn't interfere with the variable that tries to determine which number we're at

[drl]

Hi.

This output:

Code: Select all

% ./s1
 total = 23
 trash = 10

 Current state:
drafts:
#1#  ,01  ,1  ,44fox  ?1

inbox:
,1  ,7  ,8  ,9  1  7  8  9

outbox:
,01  ,02  ,1  ,3  ,6  ,hello,world  1  3  6  list,3

trash:
1  10  15  16  17  3  6  7  8  9


 Final   state:
drafts:
#1#  ,44fox  ?1

inbox:
1  7  8  9

outbox:
,hello,world  1  3  6  list,3

trash:
1  10  11  12  13  14  15  16  17  18  19  2  20  21  3  4  5  6  7  8  9

 total = 12
 trash = 21

was created by this script:

Code: Select all

#!/bin/sh

# @(#) s1       Demonstrate move and rename.

# Provide audit trail for learning and debugging.
# Interchange the two debug lines to toggle printing on and off.

debug="echo"
debug=":"

# Set up the test situation.
# For production, comment out the remove and create calls.

./remove
./create

count() {
  local a b c d total
  a=$( ls -1 drafts | wc -l )
  b=$( ls -1 inbox | wc -l )
  c=$( ls -1 outbox | wc -l )
  d=$( ls -1 trash | wc -l )
  total=$( expr $a + $b + $c )

  echo " total = $total"
  echo " trash = $d"
}
count

echo
DIRS="drafts inbox outbox"
echo " Current state:"
ls -R $DIRS trash

echo
$debug " audit trail of operations:"
$debug

highest=0
for d in $DIRS
do
  for file in $( ls -1 $d/ | grep '^,[0-9][0-9]*$' )
  do
    $debug " Working on $d/$file"
    n=${file#,}
    $debug " extracted :$n: from :$file: in :$d:"
    m=$( expr $n + 0 )
    $debug " transformed $n into $m"
    xform=$m
    if [ -f trash/$xform ]
    then
      highest=$( expr $highest + 1 )
      $debug " set highest number to :$highest:"
      while [ -f trash/$highest ]
      do
        $debug " highest increased to $highest"
        highest=$( expr $highest + 1 )
      done
      mv $d/$file trash/$highest
      $debug " moved (bumped) $d/$file to trash/$highest"
    else
      # echo mv $d/$xform trash
      mv $d/$file trash/$xform
      $debug " moved $d/$file to trash/$xform"
      highest=$xform
      $debug " set highest number to :$highest:"
    fi
  done
done

echo
echo " Final   state:"
ls -R $DIRS trash

echo
count

exit 0

It does not over-write extant files in trash, it fills in the holes in the numeric sequence of files in trash, and it prints a count before and after so that you can see the "conservation of files" ... cheers, drl

[Blyiss]

Thank you, guys, you are great people. Both scrips, dri's final reduction and Grifter's, are excellent.
I didn't have enough of emails and I made just empty files in metacity that obviously uses gedit as its default editor that leaves hidden backup files with the "~" ending. They are not seen in metacity and gnome-commander which I usually use, and I completely forgot about this gedit behavior. That's why there appeared errors. But now everything is great! Would you mind if at some point I will publish (though they are already published, but anyway) your scripts at my web-site so that other people could use them, since nothing like what you did is available for MH?
Can you also help me to make your script automatic so that I set and forget about it? I have a script from wiki for getmail to check mails every 30 sec. How to include your scrips into it?

Code: Select all

while sleep $SLEEP; do
        wait # Wait for previous fetch to complete

        cmd="getmail -d"
        for file in `find ~/.getmail -regex '.*/getmailrc-[a-zA-Z0-9-]*$'`; do
                 cmd="$cmd"" --rcfile ""$file"
        done
        echo $cmd
        $cmd &
        echo "Sleeping for $SLEEP seconds"
done