DokuWiki

I wrote this to, (A) practice writing a script with the “scp” command, and (B) upload my “code” directory to my docuwiki installation so that it could be web-browsable. This is my first “real” script that I'm putting 'into the wild' per se, so have some mercy if it doesn't work too well :)

Note also that it's completely un-cross-platform. It assumes you're running Linux both on the client and the server, although I think that with some modification it may be runnable from a Windows client. The main roadblock is that Windows doesn't have an “SCP” command, and although I believe Python can use a module to open an SCP connection, I wanted to use the built-in one in case the user has individual settings for ssh-agent or public-key authentication. I don't have a WinXP machine to test on, so if someone wants to do that, feel free.

Also, the script suffers from a number of limitations: All pages must be lowercase, which effectively breaks Java files if you use the filename from DocuWiki when saving them back to disk (they should still be viewable, of course). This limitation is actually from within DocuWiki itself. Also, the only code files supported are those for Python, Java, and C. I use mostly Python, some Java, and have just started C, which is why those 3 are supported. It should be relatively straightforward to add more; see the source code and add the appropriate extensions.

These issues mean it's probably more useful as a showcase of source code than a way to collaborate on them. There's always patch for that.

To use this script, customize the constants USER_AT_HOST, DIRECTORY_TO_UPLOAD, and DOCUWIKI_INSTALL_DIRECTORY.

Okay, here goes:

#!/usr/bin/python
#Adam Gomaa 2006. Started on October 24th. 
# A version 0.2 of the DocuWikiUpload script
# Everything except the remote directory structure working on November 3rd.
 
"""Mirrors, formats, and uploads a directory of source code files into a
Docuwiki installation.
 
This 0.2 has been modularized. Please note that there is no provision for
passwords; the SCP command should prompt you if one is necessary. Also note
that this will upload files with a .docuwiki extension which will _not_ show up
in docuwiki until the extensions have been changed to .txt. This was done to
avoid thorny issues with unintentionally deleting .txt files in the cleanUp()
module. 
 
The extension renaming can be done with a simple shell script:
    for file in *.docuwiki ; do mv $file `echo $file | sed 's/\(.*\.\)txt/\1docuwiki/'` ; done
Run this in the docuwiki data/pages/ directory. 
I've decided not to do the renaming from within this script, because that would
involve opening a remote shell or downloading/renaming/uploading over a remote
connection, both of which seem like overkill to me.
 
Lastly, you'll notice that os.walk is called several times. This needs to be 
done because the first function changes the results of os.walk, so we need 
to make sure it is performed again. 
 
Copyright (C) 2006  Adam Gomaa
GNU GPL version 2 or greater.
 
This program is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License
as published by the Free Software Foundation; either version 2
of the License, or (at your option) any later version.
 
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.
 
You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
 
"""
import os
 
USER_AT_HOST="user@host.com"
DIRECTORY_TO_UPLOAD="/home/user/code/"
DOCUWIKI_INSTALL_DIRECTORY="~/host.com/docuwiki/"
 
#Copyright notice
print "This is Free software licensed under the GNU GPL."
print "See included source for copyright details."
 
 
###Functions
def mirrorCodeDir():
 
    def copyFiles(name,root):
        #Open and read the file...
        readFile=file(os.path.join(root,name),'r')
        readData=readFile.read()
        readFile.close()
 
 
        writeFile=file(os.path.join(root,name.lower())+".docuwiki",'w')
        #... then write to (file).docuwiki according to what kind of source
        #code file it is. 
        #The weird concatenation is so that </c0de> doesn't stop
        #this script from showing correctly in docuwiki. 
        if ".py" in name:
            writeFile.write("<co"+"de python>\n"+readData+"\n</co"+"de>")
        elif ".java" in name:
            writeFile.write("<co"+"d:e java>\n"+readData+"\n</co"+"de>")
        elif ".pl" in name:
            writeFile.write("<co"+"de perl>\n"+readData+"\n</co"+"de>")
        elif ".c" in name or ".h" in name:
            writeFile.write("<co"+"de c>\n"+readData+"\n</co"+"de>")
        #Additional source code file extensions can be included here. 
        #be sure to exclude binary files, you don't want those to show up in
        #docuwiki. 
        writeFile.close()
 
 
    walkReturn=os.walk(DIRECTORY_TO_UPLOAD)
    for root, dirs, files in walkReturn:
        for name in files:
            # This should probably use a regular expression. 
            if ('.py' in name or '.java' in name or '.c' in name or ".h" 
in name or ".pl" in name) and (".docuwiki.txt" not in name and ".class"
not in name and ".pyc" not in name and "~" not in name and ".bak" not in name):
                copyFiles(name,root)
 
 
def makeRemoteDirStructure():
    def generateList():
        mkdirlist=[]
        walkReturn=os.walk(DIRECTORY_TO_UPLOAD)
        for root, dirs, files in walkReturn:
            for directory in dirs:
                mkdirlist.append(os.path.join(root,directory)[len(
DIRECTORY_TO_UPLOAD):])
        return mkdirlist
    generateList()
    #need to use os functions to create the directories. They need to be made
    #in a particular order, eg, dir/ needs to be made before dir/subdir/    
 
 
def uploadFiles():
    walkReturn= os.walk(DIRECTORY_TO_UPLOAD)
    filelist=[]
    for root, dirs, files in walkReturn:
        for name in files:
            try:
                if name[-9:]=='.docuwiki':
                    filelist.append(os.path.join(root,name))
            except IndexError:
                    pass
    commandString=''
    for each in filelist:
        commandString=commandString+" "+each
    commandString="scp "+commandString+' '+USER_AT_HOST
    commandString=commandString+":"+DOCUWIKI_INSTALL_DIRECTORY+"data/pages/"
    os.system(commandString)
 
 
def cleanUp():
    walkReturn= os.walk(DIRECTORY_TO_UPLOAD)
    """Iterates through the code directory, checks for files ending
    in the .docuwiki.txt, and removes them. 
    """
    removedFileCount=0
    for root, dirs, files in walkReturn:
        for name in files:
            if name[-9:]=='.docuwiki':
                removedFileCount+=1
                os.remove(os.path.join(root,name))
    print "Removed",removedFileCount,"files."
 
 
def main():
    userInput1=raw_input("Mirror "+DIRECTORY_TO_UPLOAD+'?\n')
    if 'y' in userInput1:
        print "Mirroring "+DIRECTORY_TO_UPLOAD+"...",
        mirrorCodeDir()
        print 'done'
    userInput2=raw_input("Upload files?\n")
    if 'y' in userInput2:
        print "Uploading files...",
        uploadFiles()
        print 'done'
    userInput3= raw_input("Delete .docuwiki files?\n")
    if 'y' in userInput3:
        print "Cleaning .docuwiki files..."
        cleanUp()
    #makeRemoteDirStructure() #Not yet working, disabled for now. 
###Main
if __name__=="__main__": 
    main()