The main class of the Crowd-Powered-Sort program. More...

Collaboration diagram for myShell.MainClass:

Static Public Member Functions
static void	main (String args[])
	Main method of this program. <../..>
static void	save (Boolean finished)
	Saves the results, either intermediary ones if `finished` is `false` or final ones if it is `true`. <../..>
static void	initialize (String dbPasswd)
	Initializes the classes necessary for a splitsort iteration and checks whether the sort must go on or not. <../..>
static void	generateJob (String dbPasswd)
	Performs the first half of a quicksort iteration, generates the comparisons it requires and submits them in a new job at CrowdFlower. <../..>
static void	useResults (String dbPasswd)
	Human made results are downloaded, interpreted, ameliorated and put in the myDataBases.MySQLbase instance `hitBdd` allowing the second half of a quicksort iteration to be performed. <../..>
static void	sqlFile ()
	Creates a .sql file allowing the creation of the database and the tables necessary for CPS. <../..>
static void	displayHelp ()
	Displays this program's help and exit. <../..>
Static Public Attributes
static LogWriter	log
	A LogWriter instance used to... <../..>
static Map< String, String >	config
	A map containing the data within the `config.xml` file. <../..>
static MySQLbase	hitBdd
	The myDataBases.MySQLbase instance used to store the job results, both raw and refined. <../..>
static MediaBase	mediaBdd
	A myDataBases.MediaBase used to store the content of the `media.xml` file. <../..>
static List< Integer[]>	allMedia
	A list of array containing all the media to sort and the advancement of the sort. <../..>
static CrowdManager	theBoss
	The crowdUser.CrowdManager instance supposed to be the interface with Crowdflower and the Antechamber during the splitsort. <../..>
static Interpreter	inter
	This Interpreter will read the `config.xml` file and write the `results.xml` one. <../..>
static Comparator	comp
	The sorter.Comparator instance that will perform the SplitSort along each specified axis. <../..>
static Boolean	keepGoing
	Is `true` if the sort must continue; is `false` when everything is over. <../..>

Detailed Description

The main class of the Crowd-Powered-Sort program.

This class contains the main method that parses command line arguments to call the right function. It also implements :

The displaying of a quick help for the user if he asks for it or if no arguments were used.
The creation of a sql file containing the information necessary to create the DBcps database used by this program.

Author:: Leo Perrin (perrin.leo@gmail.com)

Definition at line 29 of file MainClass.java.

Member Function Documentation

static void myShell.MainClass.displayHelp ( ) [static]

Displays this program's help and exit.

The appearance of the help displayed is copied from typical UNIX man pages.

Definition at line 464 of file MainClass.java.

      {
            System.out.println("\n\n\n =========================== CROWD-POWERED-SORT =========================== ");
            System.out.println();
            System.out.println("Sorts media along specified axis and returns the results in a file (xml format).");
            System.out.println("A CrowdFlower (<http://crowdflower.com>) account is necessary.");
            System.out.println();
            System.out.println(" OPTIONS");
            System.out.println();
            System.out.println("    --new 'DBpasswd'  Launches the splitsort of the media contained in the ");
            System.out.println("                      data/media.txt file or in the data/media folder (this must be ");
            System.out.println("                      precised in the config.xml file). 'DBpasswd' is the password of");
            System.out.println("                      the database. In order for the sort to continue, you must use");
            System.out.println("                      this program again, but with the '--iteration' option when you");
            System.out.println("                      receive an e-mail.");
            System.out.println();
            System.out.println("    --iteration 'DBpasswd'  Works just like '--sort' but launches a unique ");
            System.out.println("                      splitsort iteration. Results are stored as xml files");
            System.out.println("                      corresponding to each axis in the '../data/tmp' folder.");
            System.out.println();
            System.out.println("    --sql-file  creates a .sql file which, uploaded with the mysql root account,");
            System.out.println("                will create the database DBcps and the tables necessary for CPS");
            System.out.println("                to work, i.e 'hit' and 'results'.");
            System.out.println();
            System.out.println("    --check=[OPTION]  this option can have several value :");
            System.out.println("          config            checks the content of the config.xml file.");
            System.out.println("          sql-db 'DBpasswd' checks the connection with the MySQL database.");
            System.out.println("          newjob      attempts to create a new job by copying the one whose id");
            System.out.println("                      is specified in the config.xml file");
            System.out.println("          final-email Asks the antechamber to send a final notification email");
            System.out.println("                      spotting the end of an imaginary splitsort. Fake results are");
            System.out.println("                      attached in order to check this as well.");
            System.out.println();
            System.out.println("    --help displays this help and exit");
            System.out.println("\n\n\n");
      }

static void myShell.MainClass.generateJob ( String dbPasswd ) [static]

Performs the first half of a quicksort iteration, generates the comparisons it requires and submits them in a new job at CrowdFlower.

First, this class' attributes are set using the initialise method.
The Comparator instance performs the first step of a quicksort iteration and thus generates a list of unknown comparisons.
These comparisons are sent to a CrowdManager instance.
If it is necessary (i.e, if there ARE unknown comparisons), the CrowdManager instance then generates the HITs and uploads them to the right job at CrowdFlower.

Parameters:

dbPasswd The password of the database matching the user specified in the config.xml file.

Definition at line 336 of file MainClass.java.

      {
            log.append("\n___________ generating Jobs  ___________");
            
            // step 1: initialization
            initialize(dbPasswd);
            if (keepGoing)
            {
                  
                  // step 2: first step of the quicksort
                  comp.sortIteration();
                  log.append("[DONE] first step of the quicksort");
                  
                  // step 3: sending unknown comparisons to a CrowdManager instance
                  comp.demandNewComparisons(theBoss.getGreater(),theBoss.getSmaller(),theBoss.getAxes());
                  log.append("[DONE] asking for new comparisons\n   Note : " +
                    theBoss.getGreater().size() + " new comparisons required") ;
                  
                  // step 4: uploading a new job if necessary
                  if (theBoss.getGreater().size() >= 1)
                  {
                        Integer jobId = theBoss.createNewJob();
                        Interpreter.setLastJob(jobId);
                        log.append("[DONE] creating a new job at CrowdFlower (id=" + jobId + ")");
                        theBoss.uploadHIT(jobId);
                        log.append("[DONE] uploading HIT\n   ... Now waiting for HIT results.");
                  }
                  else
                  {
                        log.append("   Note : all comparisons are already known : no HIT");
                        useResults(dbPasswd);
                  }
            }
      }

static void myShell.MainClass.initialize ( String dbPasswd ) [static]

Initializes the classes necessary for a splitsort iteration and checks whether the sort must go on or not.

In this method, several classes are initialized using the content of the media.xml and config.xml files. They correspond to all of the MainClass attribute.

It also checks whether the sort should go on using the Comparator.notFinished( ) method.

Parameters:

dbPasswd The password of the MySQL database used.

Returns:: true if the sort should continue, false if it is already over.

Definition at line 269 of file MainClass.java.

      {
            config = Interpreter.getConfig() ;
            log.append("[DONE] reading configuration file");
            
            mediaBdd = new MediaBase(config.get("mediaPath"));
            log.append("[DONE] reading media to be sorted");
            
            inter = new Interpreter(mediaBdd);
            log.append("[DONE] initialisation of the Interpreter");
            
            Integer drop = Integer.parseInt(config.get("drop"));        // the size of the arrays under which they are not sorted
            log.append("[DONE] array to be sorted created\n   Note : the sort will stop for arrays having" +
                        " a smaller than " + drop + " size");
            
            Map<String,SplitSort> splitsortMap = new HashMap<String,SplitSort>();
            List<String> axes = Interpreter.getAxes();
            
            keepGoing = false;      // "default" value
            for (String axis : axes)
            {
                  allMedia = inter.getCurrentState(log,axis);
                  SplitSort sorter = new SplitSort(allMedia,drop);
                  splitsortMap.put(axis,sorter);
                  if (sorter.notFinished())
                        keepGoing = true;
            }
            
            // If the sort must go on :
            if (keepGoing)
            {
                  hitBdd = new MySQLbase("jdbc:mysql://localhost:3306/" + config.get("sqlBase") , config.get("sqlUser"),
                              dbPasswd, Interpreter.getFields("SQLfield"), Interpreter.getFields("SQLtype"));
                  log.append("[DONE] connexion to mySQL database");
                  
                  comp = new Comparator(splitsortMap,hitBdd);
                  log.append("[DONE] initialisation of the comparator");
                  
                  theBoss = new CrowdManager(Integer.parseInt( config.get("cfJobNumber") ), config.get("cfKey"),
                              config.get("antechamberUrl"), hitBdd, mediaBdd, Interpreter.getFields("CSVfield") );
                  log.append("[DONE] initialisation of the CrowdManager");
            }
            // Otherwise, final results are written without going further.
            else
            {
                  comp = new Comparator(splitsortMap);
                  log.append("[DONE] initialisation of the comparator.\n   Note: No connexion to the My SQL database created.");
                  save(true);
            }
      }

static void myShell.MainClass.main ( String args[] ) [static]

Main method of this program.

Parses command line option to launch the right function. If no argument is specified, help is displayed using the displayHelp() method. A description of what is done depending on the command line arguments is given in the body of the displayHelp() method.

Parameters:

args	Arguments sent using the command line when the program is launched.

Definition at line 86 of file MainClass.java.

      {
            // creating a new log file
            log = new LogWriter();
            
            // if no argument was given, displays help
            if (args.length == 0)
                  displayHelp();
            
            // if at least one argument is given, it depend on its value:
            else if (args.length > 0)
            {
                  // displaying help
                  if ( args[0].equals("--help") )
                        displayHelp();

                  // creating the .sql file
                  else if ( args[0].equals("--sql-file") )
                        sqlFile();
                  
                  // creating a new splitsort, but checking the config file and the connection with the database first.
                  else if ( args[0].equals("--new") )
                  {
                        if (args.length <= 1)
                              displayHelp();    // if no password was given, displays help
                        else if (Interpreter.checkFile(log))
                              if ( MySQLbase.testConnection(log, args[1]) )
                                    generateJob(args[1]);   // if all the tests were correct, a job is generated
                        else
                              System.out.println("\nERROR : a problem was encountered. Check logs for more details.");  
                  }
                  
                  // launching one iteration, but checking the config file and the connection with the database first.
                  else if ( args[0].equals("--iteration") )
                  {
                        if (args.length <= 1)
                              displayHelp();    // if the password was not given, displays help
                        else if (Interpreter.checkFile(log))
                              if ( MySQLbase.testConnection(log, args[1]) )   // if the tests were correct:
                              {
                                    useResults(args[1]);    // results are used to finish a quicksort iteration
                                    comp.display();
                                    if (keepGoing)
                                          generateJob(args[1]);   // a new one is launched and a new job is created if it is not finished
                              }
                        else
                              System.out.println("\nERROR : a problem was encountered. Check logs for more details.");
                        else
                              System.out.println("\nERROR : a problem was encountered. Check logs for more details.");
                  }
                   
                  
                  // checking...
                  else if (args[0].equals("--check"))
                  {
                        // ... nothing (not enough arguments)
                        if (args.length == 1)
                              displayHelp();
                        
                        // ... the content of the configuration file
                        if (args[1].equals("config"))
                        {
                              if (Interpreter.checkFile(log))
                                    System.out.println("\nThe config.xml file contains all the information necessary.\n\n");
                              else
                                    System.out.println("\nA problem was encountered (information missing)." +
                                                "Check log for more information\n");
                        }
                        
                        // ... the connection with the database
                        else if (args[1].equals("sql-db"))
                        {
                              if (args.length <= 2)
                                    displayHelp();    // if the password was not given, displays help
                              else if ( MySQLbase.testConnection(log, args[2]) )
                                    System.out.println("connection with the database successfully created");
                              else
                                    System.out.println("A problem was encountered while connecting with the database." +
                                          "Check log for more information\n");
                        }

                        // ... the creation of a HIT
                        else if (args[1].equals("newjob"))
                        {
                              config = Interpreter.getConfig() ;
                              CrowdManager test = new CrowdManager(Integer.parseInt( config.get("cfJobNumber") ), config.get("cfKey"),
                                          config.get("antechamberUrl"), null, null, Interpreter.getFields("CSVfield") );
                              System.out.println(" new job identifier : " + test.createNewJob());
                        }
                        
                        // ... the sending of a final e-mail when everything is finished
                        else if (args[1].equals("final-email"))
                        {
                              config = Interpreter.getConfig();
                              CrowdManager.thisIsTheEnd(config.get("antechamberUrl") + "main.php",
                                          "<?xml version='1.0' encoding='UTF-8'?>\n" +
                                          "<results>\n  <media id='99'>\n    <timecode>99</timecode>\n    <name>truc99.ogg</name>\n" +
                                          "    <axe1>-0.8181818181818181</axe1>\n  </media>\n</results>");
                              System.out.println("Signal sent at " + config.get("antechamberUrl") + "main.php\n");
                        }
                        
                        // ... nothing because the argument given was invalid: displays help
                        else
                              displayHelp();
                  }
                  
                  // displaying help otherwise
                  else
                        displayHelp();
            }
            
            // writing the date and time of end in the log file.
            log.finish();
            System.out.println();
      }

static void myShell.MainClass.save ( Boolean finished ) [static]

Saves the results, either intermediary ones if finished is false or final ones if it is true.

If the finished parameter is false, then the current state of the splitsort iswritten in as much < axis>.xml files as there are axes, with < axis>'s value going through all the axes considered. Such results are formatted so as to be easily retrieved during the next iteration.

If finished is true, final results are saved in the ../results.xml file. It contains all the information in the media.xml file plus the coordinates of each one of them along each axis. When the file is written, it is sent to the Antechamber using . Such call should only be made when the sort is done.

Parameters:

finished If true, write definitive results ; if false, write temporary ones.

See also:: Interpreter.generateIntermediaryXMLs(String, List); Interpreter.generateFinalXML(String, Map); crowdUser.CrowdManager.thisIsTheEnd(String, String)

Definition at line 221 of file MainClass.java.

      {
            if (finished)
            {
                  Map<String,Integer[]> xmlParameters = new Hashtable<String,Integer[]>();
                  // writing results in the shell, formatted like a python dictionnary to make it easier to check it.
                  System.out.println("This is for '../data/CM-stub/stub.py check' if you where testing CPS:");
                  System.out.print("{ ");
                  for ( Iterator<String> str = comp.getSplitSorts().keySet().iterator(); str.hasNext(); )
                  {  
                        String axis = str.next();
                        Integer[] res = comp.getSplitSorts().get(axis).getFinalResult();
                        System.out.print(" '" + axis + "' : [");
                        for (int i=0; i<res.length; i++)
                              System.out.print(res[i]+", ");
                        System.out.print("] ,  ");
                        xmlParameters.put(axis, res);
                  }
                  System.out.print("}\n\n");
                  // saving results in the ../data/results.xml file
                  String results = inter.generateFinalXML("../results.xml",xmlParameters);
                  log.append("[DONE] writing results in results.xml");
                  // sending final email
                  CrowdManager.thisIsTheEnd(config.get("antechamberUrl") + "main.php",results);
                  log.append("[DONE] sending end notification e-mail.");
                  System.out.println("\n\nThis splitsort is finished! You can now get the results in the data/results.xml file.");
            }
            else
                  for (String axis : comp.getSplitSorts().keySet())
                  {
                        log.append("[DONE] writing temporary state in " + axis + ".xml");
                        inter.generateIntermediaryXMLs("../data/previously/" + axis + ".xml",comp.getCurrentState().get(axis));
                  }
      }

static void myShell.MainClass.sqlFile ( ) [static]

Creates a .sql file allowing the creation of the database and the tables necessary for CPS.

It contains the information necessary to create the hit and results table, each one having the fields necessary for the CPS to work as well as those added in the config.xml file. Each table is creating in a different way:

hit's content depends completely of the content of the config.xml file ('hardDataFields' node)
results contains axis, idMedia1, idMedia2, greater, nameMedia1, nameMedia2 and what is in the 'resultsFields' node of the config.xml file.

The file generated is ../data/mysql/createCPStables.sql.

Definition at line 431 of file MainClass.java.

      {
            String user = Interpreter.getConfig().get("sqlUser");
            String fileContent = "CREATE DATABASE DBcps ;\n" +
                        "GRANT DELETE,DROP,SELECT,INSERT,LOCK TABLES,UPDATE,CREATE ON DBcps.* TO '" +
                        user + "'@'localhost' ;\n" +
                        "USE DBcps ;\n" +
                        "CREATE TABLE results (axis VARCHAR(40), idMedia1 INT, idMedia2 INT, greater TINYINT," +
                        " nameMedia1 VARCHAR(300), nameMedia2 VARCHAR(300) ) ;\n" +
                        "CREATE TABLE hit (";
            List<String> fields = Interpreter.getFields("SQLfield");
            List<String> types = Interpreter.getFields("SQLtype");
            for (int i=0; i<fields.size()-1; i++)
                  fileContent += fields.get(i) +" "+ types.get(i) +", ";
            fileContent += fields.get(fields.size()-1) +" "+ types.get(fields.size()-1) +" ) ;";
            try
            {
                  BufferedWriter sqlFile = new BufferedWriter
                              (new FileWriter("../data/mysql/createCPStables.sql",false));
                  sqlFile.write(fileContent);
                  sqlFile.newLine();
                  sqlFile.close();
            }
            catch (Exception e) { e.printStackTrace(); }
            System.out.println("File created : data/mysql/createCPStables.sql");
      }

static void myShell.MainClass.useResults ( String dbPasswd ) [static]

Human made results are downloaded, interpreted, ameliorated and put in the myDataBases.MySQLbase instance hitBdd allowing the second half of a quicksort iteration to be performed.

The MainClass' attributes are initialized.
Results are read and ameliorated using code from P. Ipeirotis and is students' program, get another label.
The Comparator instance performs the last step of the quicksort.
A dump of the DBcps database is created to make sure that, even if this program fails, it won't be necessary to submit again already sent HITs.
The results are saved along each axis in an xml file.

Definition at line 388 of file MainClass.java.

      {
            log.append("\n___________ Using Results  ___________");
            
            // step 1: initialization
            initialize(dbPasswd);
            if (keepGoing)
            {
                  comp.sortIteration();   // Each splitsort recreates its comparisons in order to be able to understand the results
                  String path2results = theBoss.getHITresults( Interpreter.getLastJob() );
                  
                  // step 2 : after the Job is done, the results are read and ameliorated.
                  theBoss.readHITresults( path2results );
                  log.append("[DONE] reading HIT results");
                  
                  // step 3 : last step of the quicksort
                  comp.endIteration();
                  log.append("[DONE] last step of the quicksort");
                  
                  // step 4 : generating a dump of the database
                  hitBdd.generateDump();
                  log.append("[DONE] dump of the MySQL database");
                  
                  // step 5 : saving current results
                  save(false);
            }
      }