Trying to remove excess entries from job Config History caused by envInject

[Bailey, Darragh]

Hi,

I'm trying to put together a script that can be run via the console or
possibly the groovy plugin, to remove all the additional saved configs
added due to how the envInject and jobConfigHistory plugins interact.

Related Jira is https://issues.jenkins-ci.org/browse/JENKINS-14144.

The core problem is the addition/removal of the xml element
"<org.jenkinsci.plugins.envinject.EnvInjectListener_-JobSetupEnvironmentWrapper/>"
from the buildWrappers node that occurred at the start/end of each build
once the envInject plugin was being used by the build.

What I was hoping to do, was compare the xml contents of each saved
config, ignore the xml node created by envInject and then remove any
saved configs that were identical.

So I've cobbled together something that mostly works (pasted at the end)
from various webpages and posts online. It uses an xsl transform to
discard nodes matching
"org.jenkinsci.plugins.envinject.EnvInjectListener_-JobSetupEnvironmentWrapper",
and then normalizes the xml.

For some reason though, when I ask the "Document" object to do the
comparison using the 'isEqualNode' method, there are a number of cases
where it says the xml does not match, but if I compare the xml by
formatting it and outputting to strings, it does match.

I'm a little stumped as to what would be getting picked up as being
different. Is there something that the transform would be missing?

Hoping that someone here might be able to point me in the right
direction, and that the code below helps others hitting that JIRA issue.



Here's some output from a run of the script:
Ignoring as files differ: config-history/2012-08-13_11-17-31/config.xml
config-history/2012-08-13_12-29-07/config.xml
Ignoring as files differ: config-history/2012-08-13_12-29-07/config.xml
config-history/2012-08-13_18-51-04/config.xml
Ignoring as files differ: config-history/2012-08-13_18-51-04/config.xml
config-history/2012-08-13_18-51-05/config.xml
XML output is equal
Following file can be deleted: config-history/2012-08-13_18-51-05/config.xml
Ignoring as files differ: config-history/2012-08-13_18-51-04/config.xml
config-history/2012-08-16_19-34-46/config.xml
Ignoring as files differ: config-history/2012-08-16_19-34-46/config.xml
config-history/2012-08-16_19-34-51/config.xml
Ignoring as files differ: config-history/2012-08-16_19-34-51/config.xml
config-history/2012-08-16_19-35-38/config.xml
Ignoring as files differ: config-history/2012-08-16_19-35-38/config.xml
config-history/2012-08-16_19-35-49/config.xml
XML output is equal <--- previous line says that the check using the
Document object says they are not the same, why?
Following file can be deleted: config-history/2012-08-16_19-35-49/config.xml
Files are the same: config-history/2012-08-16_19-35-38/config.xml
config-history/2012-08-16_19-35-50/config.xml
Following file can be deleted: config-history/2012-08-16_19-35-50/config.xml
Ignoring as files differ: config-history/2012-08-16_19-35-38/config.xml
config-history/2012-08-16_19-41-12/config.xml
XML output is equal
Following file can be deleted: config-history/2012-08-16_19-41-12/config.xml


A direct textual comparison of the two files
(config-history/2012-08-13_18-51-04/config.xml,  on disk:
diff -u config-history/2012-08-13_18-51-04/config.xml
config-history/2012-08-13_18-51-05/config.xml
--- config-history/2012-08-13_18-51-04/config.xml    2012-08-13
18:51:04.592635936 +0000
+++ config-history/2012-08-13_18-51-05/config.xml    2012-08-13
18:51:05.592648870 +0000
@@ -207,6 +207,5 @@
<postSuccessfulBuildSteps/>
<postFailedBuildSteps/>
</hudson.plugins.release.ReleaseWrapper>
-
<org.jenkinsci.plugins.envinject.EnvInjectListener_-JobSetupEnvironmentWrapper/>
</buildWrappers>
</project>
\ No newline at end of file


Changing the script below to output the strings to the script console,
also shows them to be identical.



-----------------------------------
import hudson.plugins.jobConfigHistory.*;

import javax.xml.parsers.*;
import javax.xml.transform.*;
import javax.xml.transform.dom.*;
import javax.xml.transform.stream.*;

import org.apache.commons.lang.StringUtils;
import org.apache.xml.serialize.OutputFormat;
import org.apache.xml.serialize.XMLSerializer;

import org.w3c.dom.*;
import org.xml.sax.*;

// see
http://stackoverflow.com/questions/141993/best-way-to-compare-2-xml-documents-in-java
public class BasicXsl {
     // applies the xsl string to inFilename and returns the result as a
string
     public static String xsl(String inFilename, String xsl) {
         try {
             Source source = new StreamSource(new
FileInputStream(inFilename));
             StringWriter writer = new StringWriter();

             // Create transformer factory
             TransformerFactory factory = TransformerFactory.newInstance();
             Templates template = factory.newTemplates(new
StreamSource(new StringReader(xsl)));

             Transformer xformer = template.newTransformer();
             xformer.transform(source, new StreamResult(writer));

             return(writer.toString());
         } catch (FileNotFoundException e) {
         } catch (TransformerConfigurationException e) {
             // An error occurred in the XSL file
         } catch (TransformerException e) {
             // An error occurred while applying the XSL file
             // Get location of error in input file
             SourceLocator locator = e.getLocator();
             int col = locator.getColumnNumber();
             int line = locator.getLineNumber();
             String publicId = locator.getPublicId();
             String systemId = locator.getSystemId();
         }
         return null;
     }
}

public class trimpath {
   public static String shortpath(String path, int elements) {
     String [] filePathArray = path.split(File.separator);
     return StringUtils.join(Arrays.copyOfRange(
           filePathArray, filePathArray.length - elements,
filePathArray.length), File.separator)
   }
}

// see
http://stackoverflow.com/questions/1137563/xsl-how-to-copy-a-tree-but-removing-some-nodes/1137628#1137628
//
// simple xsl template to discard any node matching
//
org.jenkinsci.plugins.envinject.EnvInjectListener_-JobSetupEnvironmentWrapper
xsl = "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n" +
       " <xsl:stylesheet version=\"1.0\"
xmlns:xsl=\"http://www.w3.org/1999/XSL/Transform\">\n" +
       " <xsl:template match=\"node() | @*\">\n" +
       " <xsl:copy>\n" +
       " <xsl:apply-templates select=\"node() | @*\"/>\n" +
       " </xsl:copy>\n" +
       " </xsl:template>\n" +
       " <xsl:template
match=\"org.jenkinsci.plugins.envinject.EnvInjectListener_-JobSetupEnvironmentWrapper\"/>
<!-- this empty template will this element -->\n" +
       "</xsl:stylesheet>";


// see
http://stackoverflow.com/questions/141993/best-way-to-compare-2-xml-documents-in-java
//
// Set up docbuilder stuff for xml parsing/comparison
dbf = DocumentBuilderFactory.newInstance();
dbf.setNamespaceAware(true);
dbf.setCoalescing(true);
dbf.setIgnoringElementContentWhitespace(true);
dbf.setIgnoringComments(true);
db = dbf.newDocumentBuilder();

found = 0;

for(item in Hudson.instance.getItems(hudson.model.Project)) {

   AbstractProject project = item;
   println(item.getName());
   jch = new JobConfigHistoryProjectAction(project);
   allJch = jch.getConfigs()
   if(allJch.size() < 1) {
     continue;
   }

   // avoid deleting while the build is running, just in case
   if(item.isBuilding()) {
     continue;
   }

   // note: configs are ordered latest to earliest
   itr = allJch.listIterator(allJch.size());

   // always keep the very first one
   prevCfg = itr.previous();
   prevCfgFile = URLDecoder.decode(prevCfg.getFile(), "utf-8") +
"/config.xml"
   prevdoc = db.parse(new InputSource(new
StringReader(BasicXsl.xsl(prevCfgFile, xsl))));
   prevdoc.normalizeDocument();

   while(itr.hasPrevious()) {
     cfg = itr.previous();

     cfgFile = URLDecoder.decode(cfg.getFile(), "utf-8") + "/config.xml"
     cfgdoc = db.parse(new InputSource(new
StringReader(BasicXsl.xsl(cfgFile, xsl))));
     cfgdoc.normalizeDocument();

     // compare using document object
     if(!prevdoc.isEqualNode(cfgdoc)) {
       println("Ignoring as files differ: " +
trimpath.shortpath(prevCfgFile, 3) + " " + trimpath.shortpath(cfgFile, 3) );

       // see
http://stackoverflow.com/questions/139076/how-to-pretty-print-xml-from-java
       //
       // since the above check doesn't appear to catch all contents
that should be considered equal
       // compare the actual xml outputted
       format = new OutputFormat(cfgdoc);
       format.setLineWidth(65);
       format.setIndenting(true);
       format.setIndent(2);
       out1 = new StringWriter();
       serializer = new XMLSerializer(out1, format);
       serializer.serialize(cfgdoc);

       format = new OutputFormat(prevdoc);
       format.setLineWidth(65);
       format.setIndenting(true);
       format.setIndent(2);
       out2 = new StringWriter();
       serializer = new XMLSerializer(out2, format);
       serializer.serialize(prevdoc);

       if (out1.toString() == out2.toString()) {
         // why is this? how can the formatted output be the same, but a
comparison of the xml docs
         // which is told to ignore all whitespace differences, says
they are not the same?
         println("XML output is equal");
       } else {
         prevCfg = cfg;
         prevCfgFile = cfgFile;
         prevdoc = cfgdoc;
         continue;
       }
     } else {
       println("Files are the same: " + trimpath.shortpath(prevCfgFile,
3) + " " + trimpath.shortpath(cfgFile, 3) );
     }

     // ignore those created by other users besides SYSTEM since they
were deliberately saved
     if( cfg.getUserID() != "SYSTEM" ) {
        continue;
     }
     println("Following file can be deleted: " +
trimpath.shortpath(cfgFile, 3));

     found += 1;
     // limit for now
     if (found > 10) {
       break;
     }
   }

   // limit for now
   if (found > 10) {
     break;
   }
}

--
Regards,
Darragh Bailey

[Bailey, Darragh]

Hi,

Resolved my own problem.


On 01/11/12 13:47, Darragh Bailey wrote:
> Hi,
>
> I'm trying to put together a script that can be run via the console or
> possibly the groovy plugin, to remove all the additional saved configs
> added due to how the envInject and jobConfigHistory plugins interact.
<snipped>
> For some reason though, when I ask the "Document" object to do the
> comparison using the 'isEqualNode' method, there are a number of cases
> where it says the xml does not match, but if I compare the xml by
> formatting it and outputting to strings, it does match.
>
> I'm a little stumped as to what would be getting picked up as being
> different. Is there something that the transform would be missing?

That was exactly the problem. The transform was leaving a blank line
when removing the undesired element, which only became apparent when I
decided to switch to using a simple 'for ( node in cfgDoc ) {
println(node); }' to print the xml document.


Corrected xsl template is:
<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0"
xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output indent="yes"/>
<xsl:strip-space elements="*"/>
<xsl:template match="node() | @*">
<xsl:copy>
<xsl:copy-of select="@*" />
<xsl:apply-templates/>
</xsl:copy>
</xsl:template>
<xsl:template
match="org.jenkinsci.plugins.envinject.EnvInjectListener_-JobSetupEnvironmentWrapper"/>
<!-- this empty template will remove this element -->
</xsl:stylesheet>

Output ended up having all whitespace removed including indents despite
the '<xsl:output indent="yes"/>' element, but that didn't bother me once
it was correctly able to discard any extra configs saved due to the
envInject plugin.

Script is at the end of the email for anyone else that might find it useful.

--
Regards,
Darragh Bailey



-----

import hudson.plugins.jobConfigHistory.*;

import javax.xml.parsers.*;
import javax.xml.transform.*;
import javax.xml.transform.dom.*;
import javax.xml.transform.stream.*;

import org.apache.commons.lang.StringUtils;
import org.apache.xml.serialize.OutputFormat;
import org.apache.xml.serialize.XMLSerializer;

import org.w3c.dom.*;
import org.xml.sax.*;

// see
http://stackoverflow.com/questions/141993/best-way-to-compare-2-xml-documents-in-java
public class BasicXsl {
     // applies the xsl string to inFilename and returns the result as a
string
     public static String xsl(String inFilename, String xsl) {
         try {
             Source source = new StreamSource(new
FileInputStream(inFilename));
             StringWriter writer = new StringWriter();

             // Create transformer factory
             TransformerFactory factory = TransformerFactory.newInstance();
             Templates template = factory.newTemplates(new
StreamSource(new StringReader(xsl)));

             Transformer xformer = template.newTransformer();
             xformer.transform(source, new StreamResult(writer));

             return(writer.toString().trim());
         } catch (FileNotFoundException e) {
         } catch (TransformerConfigurationException e) {
             // An error occurred in the XSL file
         } catch (TransformerException e) {
             // An error occurred while applying the XSL file
             // Get location of error in input file
             SourceLocator locator = e.getLocator();
             int col = locator.getColumnNumber();
             int line = locator.getLineNumber();
             String publicId = locator.getPublicId();
             String systemId = locator.getSystemId();
         }
         return null;
     }
}

public class trimpath {
   public static String shortpath(String path, int elements) {
     String [] filePathArray = path.split(File.separator);
     return StringUtils.join(Arrays.copyOfRange(
           filePathArray, filePathArray.length - elements,
filePathArray.length), File.separator)
   }
}

// see
http://stackoverflow.com/questions/1137563/xsl-how-to-copy-a-tree-but-removing-some-nodes/1137628#1137628
// and
http://stackoverflow.com/questions/10711023/getting-ride-of-empty-lines-after-deleting-nodes-using-xslt
//
// simple xsl template to discard any node matching
//
org.jenkinsci.plugins.envinject.EnvInjectListener_-JobSetupEnvironmentWrapper
xsl = "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n" +
       " <xsl:stylesheet version=\"1.0\"
xmlns:xsl=\"http://www.w3.org/1999/XSL/Transform\">\n" +
       " <xsl:output indent=\"yes\"/>\n" +
       " <xsl:strip-space elements=\"*\"/>\n" +
       " <xsl:template match=\"node() | @*\">\n" +
       " <xsl:copy>\n" +
       " <xsl:copy-of select=\"@*\" />\n" +
       " <xsl:apply-templates/>\n" +
       " </xsl:copy>\n" +
       " </xsl:template>\n" +
       " <xsl:template
match=\"org.jenkinsci.plugins.envinject.EnvInjectListener_-JobSetupEnvironmentWrapper\"/>
<!-- this empty template will this element -->\n" +
       "</xsl:stylesheet>";


// see
http://stackoverflow.com/questions/141993/best-way-to-compare-2-xml-documents-in-java
//
// Set up docbuilder stuff for xml parsing/comparison
dbf = DocumentBuilderFactory.newInstance();
dbf.setNamespaceAware(true);
dbf.setCoalescing(true);
dbf.setIgnoringElementContentWhitespace(true);
dbf.setIgnoringComments(true);
db = dbf.newDocumentBuilder();

for(item in Hudson.instance.getItems(hudson.model.Project)) {

   AbstractProject project = item;
   println(item.getName());
   jch = new JobConfigHistoryProjectAction(project);
   allJch = jch.getConfigs()
   if(allJch.size() < 1) {
     continue;
   }

   // avoid deleting while the build is running, just in case
   if(item.isBuilding()) {
     continue;
   }

   // note: configs are ordered latest to earliest
   itr = allJch.listIterator(allJch.size());

   // always keep the very first one
   prevCfg = itr.previous();
   prevCfgFile = URLDecoder.decode(prevCfg.getFile(), "utf-8") +
"/config.xml"
   prevdoc = db.parse(new InputSource(new
StringReader(BasicXsl.xsl(prevCfgFile, xsl))));
   prevdoc.normalizeDocument();

   keeping=0;
   deleting=0;
   while(itr.hasPrevious()) {
     cfg = itr.previous();

     cfgDir = URLDecoder.decode(cfg.getFile(), "utf-8")
     cfgFile = cfgDir + "/config.xml"
     cfgdoc = db.parse(new InputSource(new
StringReader(BasicXsl.xsl(cfgFile, xsl))));
     cfgdoc.normalizeDocument();

     // compare using document object
     if(!prevdoc.isEqualNode(cfgdoc)) {
       println("Ignoring as files differ: " +
trimpath.shortpath(prevCfgFile, 4) + " " + trimpath.shortpath(cfgFile, 4) );
       prevCfg = cfg;
       prevCfgFile = cfgFile;
       prevdoc = cfgdoc;
       keeping += 1;
       continue;
     } else {
       println("Files are the same: " + trimpath.shortpath(prevCfgFile,
4) + " " + trimpath.shortpath(cfgFile, 4) );
     }

     // ignore those created by other users besides SYSTEM since they
were deliberately saved
     if( cfg.getUserID() != "SYSTEM" ) {
        keeping += 1;
        continue;
     }
     println("Following file can be deleted: " +
trimpath.shortpath(cfgFile, 4));
     deleting += 1;
     // paranoia
     base = item.getRootDir().getCanonicalFile();
     child = (new File(cfgDir)).getCanonicalFile();

     File parentFile = child;
     while (parentFile != null) {
       if (base.equals(parentFile)) {
         println("Deleting: " + cfgDir);
         // when happy withe text output, uncomment the following lines
         // File.deleteDir(cfgDir);
         break;
       }
       parentFile = parentFile.getParentFile();
     }
   }
   println("Kept " + keeping + " files, deleted " + deleting + " files");
}