Search This Blog

Wednesday, May 18, 2016

Example implementation of small automation framework for SoapUI


Basic design of the framework:

1. There is a separate Common.xml SoapUI project. With that implementation Common.xml must be placed in the same location as the actual test project of SoapUI.
2. Inside that project there is a Test Suite that contains Test Case which further contains Groovy steps.
3. In the groovy steps there can be a logic that is common (similar, the same) for multiple actions. It allows for centralizing the code a little bit and theoretically easier maintenance.
4. To use the methods inside our "libraries" we need to programatically load the Common.xml project , which is done in 'Setup script' of each SoapUI Test Case.

Code to load the Common.xml project is as follows:


5. In Common.xml let's have "SSL" Groovy and RunProcess steps, which serve for us as libraries with methods. Of course we can have many additional or different functionality depending on what specific project we do tests for. These libraries are listed and described below in one-by-one manner.


6. Below is the implementation of SSL, where we can find utility methods to programatically alter the SSL keystore in Test Step, or SSL TrustStore in the SoapUI project:



7. Below is the implementation of RunProcess, where we can find utility methods to run the shell commands. It can be used to do the message signing purpose where the signing is being done with conflicting wss4j-2.x library. That library is in conflict with wss4j-1.x library use by SoapUI.

Thursday, February 4, 2016

JMeter Custom Soap Sampler

This post is dedicated to a custom JMeter SOAP sampler having support to enclose attachments to the request

This functionality was not present in standard JMeter binary (v2.13) so had to be created.

The Maven project with source code and already built JAR is in Github under : Custom Soap Sampler Java Project

The plugin UI looks like that:


The sampler appears in sampler list as 'Custom SOAP Sampler'. It can work with one or more attachments (multiple attachments) and also without attachments, this way behaving as normal Soap sampler. As seen in the screenshot it has the following features:
- drop-down for SOAP protocol version with values 1_1 and 1_2
- URL - endpoint for the request
- checkbox 'Treat selected attachment as response'. If this is enabled then for services that in response return attachments the assertions can be made. So if the option is enabled there is a choice to locate the proper attachment in response by content type or content ID
- fields for adding attachment: browser to locate the file, checkbox 'Use relative paths', Content ID, type (values: resource, value). If resource is added then the content of the file is loaded as stream to the attachment object. If value is selected then the value from text is taken 'as is'.
- content-type. Values: auto, text/plain, text/xml, text/html, application/xml, application/gzip. If other types are required then the project needs to be updated and rebuilt.
- buttons to Add and Remove attachments

Other than that the sampler behaves just as other ordinary sampler.
Programmatic use of the plugin in JSR223 PreProcessor, for example to add, delete attachments from the code because there is exposed a getter: 'getAttachmentDefinition()':
//add attachments 
ArrayList attList = new ArrayList(); 
def att1 = ctx.getCurrentSampler().getAttachmentDefinition();
att1.attachment = new File(pathToFile); 
att1.contentID ="someContentID"; 
att1.contentType="application/xml"; //one of selections from the dropdown 
att1.type=1; //1 means resource, 2 means variable
attList.add(att1);
//confirm (set) adding all attachments 
ctx.getCurrentSampler().setAttachments(attList);
//set request body
ctx.getCurrentSampler().setXmlData(xmlRequest);

Thursday, January 7, 2016

Signing Soap message and its attachments by WSS4J to further use (for example) by SoapUI

In this post it is shown how to make SOAP message signing with use of the following Java library WSS4J version 2.1.x.

The topic is not so widely described and may be helpful for people who try to implement it but find issues when something has to be done more custom way. In our case performing signing in Java was a necessity due to limitation of SoapUI tool that particularly didn't have attachments signing. As a workaround, from high-level perspective we did it the following way:

1. Java project is being used for message signing and action is done through main function having parameters. Depending on number of parameters specific sub-action of signing is being done because we have to support different kinds of soap messages (PUSH, PULL, RECEIPT, RECEIPT_ERROR). PUSH is special message type because requires signing of not only the payload but also attachments. This was main the reason of creating the signing procedure outside of SoapUI.
The functionality is developed in Java project and by maven-shade-plugin exported to executable Java JAR file. This way it is standalone JAR that has all necessary dependencies (other dependent JARs).

2. SoapUI executes the jar by "java - jar "parameter1" "parameter2" "parameterX" like for example:



The parameters are parsed so that "|" (pipe) is special character to make a split, also the first parameter determines how to parse further parameters and what to expect. Note that this approach allows for any number of attachments. The only matter is for SoapUI to prepare appropriate parameters (based on actual state in SoapUI step, request attachments in the test step) and the Java to interpret the args[] in proper way. Here a question arises: why didn't we put the JAR to bin ext and loaded it into SoapUI classpath? The answer is the following: some WSS4j2.0 dependent library (xml-sec-2.0.5.jar) was in conflict with the library used by SoapUI (xml-sec-1.4.5.jar). Replacing SoapUI's lib with the newer xmlsec.jar resulted in that loading of our messageSigner.jar ws posible but SoapUI basic functionality stopped working. If we tackled that problem then it would be easier to work with external jar and have access to objects. Also it would run faster and there would be initially no synchronization issues on the SoapUI. side.But pressure of time and necessity to have working solution forced us to make it via "java -jar" command and running 'main' function with parameters. In the parameters please also note a path to unsigned request, because the JAR needs unsigned XML as an input. Preparing the arguments for the JAR and handling the output will be described in details in the separate article.
3. Java returns signed request as an output. This output is being placed in the request form in SoapUI step (made by Groovy). The original unsigned request is stored under variable.
4. After the request is executed the very first groovy assertion restores the original unsigned soap request. It is made that way to have it executed even if there is an error. We need to have the same test script after execution. Of course using SoapUI is not necessary and we can sign, send, assert requests through Java directly. But we had already plenty of test cases in SoapUI, so took (not optimal looking from perspective of time) decision to still utilize SoapUI tool. The recommendation is to use just Java and refrain from using tools if the team consists of people who know programming.

Below is the implementation of the Java project.
1. List of dependencies in the POM.xml:
    <dependencies>
        <dependency>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-api</artifactId>
            <version>2.1</version>
            <type>jar</type>
        </dependency>
        <dependency>
            <groupId>org.apache.wss4j</groupId>
            <type>pom</type>
            <artifactId>wss4j</artifactId>
            <version>2.1.3</version>
        </dependency>
        <dependency>
            <groupId>org.apache.wss4j</groupId>
            <artifactId>wss4j-ws-security-common</artifactId>
            <version>2.1.3</version>
            <type>jar</type>
        </dependency>
        <dependency>
            <groupId>org.apache.wss4j</groupId>
            <artifactId>wss4j-ws-security-dom</artifactId>
            <version>2.1.3</version>
            <type>jar</type>
        </dependency>
        <dependency>
            <groupId>commons-io</groupId>
            <artifactId>commons-io</artifactId>
            <version>2.4</version>
            <type>jar</type>
        </dependency>
    </dependencies>
2. Maven shade plugin configuration in POM.xml:
  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-shade-plugin</artifactId>
        <version>2.4.1</version>
        <executions>
          <execution>
            <phase>package</phase>
            <goals>
              <goal>shade</goal>
            </goals>
            <configuration>
              <transformers>
                <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                  <manifestEntries>
                    <Main-Class>com.messageSigner.security.WSSSignPushMessage</Main-Class>
                    <Build-Number>123</Build-Number>
                  </manifestEntries>
                </transformer>
              </transformers>
              <filters>
        <filter>
            <artifact>*:*</artifact>
            <excludes>
                <exclude>META-INF/*.SF</exclude>
                <exclude>META-INF/*.DSA</exclude>
                <exclude>META-INF/*.RSA</exclude>
            </excludes>
        </filter>
    </filters>
            </configuration>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>

3. Helper method for reading a file (for sure there are different libraries to read from a file, this is one of possibilities):
public static String readFile(String path) throws IOException {
 File file = new File(path);
 StringBuilder fileContents = new StringBuilder((int)file.length());
 Scanner scanner = new Scanner(file);
 String lineSeparator = System.getProperty("line.separator");
 try {
  while(scanner.hasNextLine()) {        
   fileContents.append(scanner.nextLine() + lineSeparator);
  }
  return fileContents.toString();
 } finally {
  scanner.close();
 }
}

4. Content of a AttachmentCallbackHandler.java (found in Apache documentation):
import javax.security.auth.callback.Callback;
import javax.security.auth.callback.CallbackHandler;
import javax.security.auth.callback.UnsupportedCallbackException;

import org.apache.wss4j.common.ext.Attachment;
import org.apache.wss4j.common.ext.AttachmentRequestCallback;
import org.apache.wss4j.common.ext.AttachmentResultCallback;

import java.io.IOException;
import java.util.ArrayList;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;

public class AttachmentCallbackHandler implements CallbackHandler {

    private final List<Attachment> originalRequestAttachments;
    private Map<String, Attachment> attachmentMap = new HashMap<>();
    private List<Attachment> responseAttachments = new ArrayList<>();

    public AttachmentCallbackHandler() {
        originalRequestAttachments = Collections.emptyList();
    }

    public AttachmentCallbackHandler(List<Attachment> attachments) {
        originalRequestAttachments = attachments;
        if (attachments != null) {
            for (Attachment attachment : attachments) {
                attachmentMap.put(attachment.getId(), attachment);
            }
        }
    }

    public void handle(Callback[] callbacks)
            throws IOException, UnsupportedCallbackException {
        for (int i = 0; i < callbacks.length; i++) {
            if (callbacks[i] instanceof AttachmentRequestCallback) {
                AttachmentRequestCallback attachmentRequestCallback =
                        (AttachmentRequestCallback) callbacks[i];

                List<Attachment> attachments =
                        getAttachmentsToAdd(attachmentRequestCallback.getAttachmentId());
                if (attachments.isEmpty()) {
                    throw new RuntimeException("wrong attachment requested");
                }

                attachmentRequestCallback.setAttachments(attachments);
            } else if (callbacks[i] instanceof AttachmentResultCallback) {
                AttachmentResultCallback attachmentResultCallback =
                        (AttachmentResultCallback) callbacks[i];
                responseAttachments.add(attachmentResultCallback.getAttachment());
                attachmentMap.put(attachmentResultCallback.getAttachment().getId(),
                        attachmentResultCallback.getAttachment());
            } else {
                throw new UnsupportedCallbackException(callbacks[i], "Unrecognized Callback");
            }
        }
    }

    public List<Attachment> getResponseAttachments() {
        return responseAttachments;
    }

    // Try to match the Attachment Id. Otherwise, add all Attachments.
    private List<Attachment> getAttachmentsToAdd(String id) {
        List<Attachment> attachments = new ArrayList<>();
        if (attachmentMap.containsKey(id)) {
            attachments.add(attachmentMap.get(id));
        } else {
            if (originalRequestAttachments != null) {
                attachments.addAll(originalRequestAttachments);
            }
        }

        return attachments;
    }
}
5. Content of the WSSSigner.java
import com.company.commons.FileSystemMethods;
import org.apache.commons.io.IOUtils;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.apache.wss4j.common.WSEncryptionPart;
import org.apache.wss4j.common.crypto.Crypto;
import org.apache.wss4j.common.crypto.CryptoFactory;
import org.apache.wss4j.common.ext.Attachment;
import org.apache.wss4j.common.util.AttachmentUtils;
import org.apache.wss4j.dom.WSConstants;
import org.apache.wss4j.dom.message.WSSecHeader;
import org.apache.wss4j.dom.message.WSSecSignature;
import org.w3c.dom.Document;
import javax.activation.DataHandler;
import javax.activation.FileDataSource;
import javax.xml.soap.*;
import javax.xml.transform.stream.StreamSource;
import java.io.*;
import java.security.KeyStore;
import java.security.Security;
import java.util.*;
import java.util.regex.Pattern;

public class WSSSigner {

    private static final Logger logger = LogManager.getLogger();

    private static String keystoreFile;
    private static String keystorePass;
    private static String keystoreAlias;
    private static String truststoreFile;
    private static String truststorePass;

    //create Properties for crypto
    private Properties propertiesCrypto = null;
    private static Crypto crypto;
    private static String signedSoapMessage;
    private static String requestType;

    public WSSSigner(HashMap mapKeystores) throws Exception{

        //this is required to prevent exception coming from BouncyCastle: "Caused by: java.lang.SecurityException: JCE cannot authenticate the provider BC ..."
        Security.addProvider(new org.bouncycastle.jce.provider.BouncyCastleProvider());

        keystoreFile = mapKeystores.get("keystoreFile");
        keystorePass = mapKeystores.get("keystorePass");
        keystoreAlias = mapKeystores.get("alias");
        truststoreFile = mapKeystores.get("truststoreFile");
        truststorePass = mapKeystores.get("truststorePass");

        System.setProperty("javax.net.ssl.keyStore", keystoreFile);
        System.setProperty("javax.net.ssl.keyStorePassword", keystorePass);
        //System.setProperty("javax.net.ssl.trustStore", truststoreFile);
        //System.setProperty("javax.net.ssl.trustStorePassword", truststorePass);

        this.propertiesCrypto = new Properties();
        this.propertiesCrypto.setProperty("org.apache.wss4j.crypto.provider", "org.apache.wss4j.common.crypto.Merlin");

        //quite naive but working method to recognize certificate type (only pfx and jks supported)
        String certificateExtension = keystoreFile.substring(keystoreFile.length()-3);
        if(certificateExtension.equalsIgnoreCase("pfx")) {
            this.propertiesCrypto.setProperty("org.apache.wss4j.crypto.merlin.keystore.type", "pkcs12");
        } else if (certificateExtension.equalsIgnoreCase("jks")) {
            this.propertiesCrypto.setProperty("org.apache.wss4j.crypto.merlin.keystore.type", "jks");
        } else {
            throw new Exception("certificate extension can be only jks or pfx while it was '" + certificateExtension + "'");
        }

        //=================================Extracting alias from keystore programatically========
        //certificate alias is provided in arguments but in case when there is only 1 then it is being used
        //it's not optimal but made like that to provide backward compatibility with the code in Groovy for SoapUI
        String alias = null;

        File file = new File(mapKeystores.get("keystoreFile"));
        FileInputStream is = new FileInputStream(file);
        KeyStore keystore = KeyStore.getInstance(this.propertiesCrypto.getProperty("org.apache.wss4j.crypto.merlin.keystore.type"));
        keystore.load(is, mapKeystores.get("keystorePass").toCharArray());

        int aliasCounter = 0;
        Enumeration enumeration = keystore.aliases();
        while(enumeration.hasMoreElements()) {
            alias = (String)enumeration.nextElement();
            aliasCounter++;
            logger.error("alias" + aliasCounter + " name: " + alias);
            //Certificate certificate = keystore.getCertificate(alias); //not useful here but for potential debugging
            //System.out.println(certificate.toString());
        }

        if (aliasCounter > 1) {
            logger.error(aliasCounter + " aliases exist, while expecting just one, so using the alias from external parameter: " + keystoreAlias);
            this.propertiesCrypto.setProperty("org.apache.wss4j.crypto.merlin.keystore.alias", keystoreAlias); //using external alias value, not trying to guess alias name
        } else {
            logger.error(aliasCounter + " alias exist, selecting the one resolved programatically.");
            mapKeystores.remove("alias");
            mapKeystores.put("alias", alias);
            keystoreAlias = mapKeystores.get("alias");
            this.propertiesCrypto.setProperty("org.apache.wss4j.crypto.merlin.keystore.alias", alias); //using alias just from certificate
        }
        //=======================================================================================

        this.propertiesCrypto.setProperty("org.apache.wss4j.crypto.merlin.keystore.password", keystorePass);
        this.propertiesCrypto.setProperty("org.apache.wss4j.crypto.merlin.keystore.file", keystoreFile);
        crypto = CryptoFactory.getInstance(propertiesCrypto);
    }
    
    //example main where input args are being extracted
    //here 4 types of requests wehere there: pull, receipt, error (receipt but with error status), push
    //push action is responsible for sending message with attachment to the node, that's why it is treated separately
    public static void main(String args[]) throws Exception {

        HashMap attachments = new HashMap<>();
        
        if(args.length>0)
            requestType = args[0];
        
        if (args.length == 3) {
            if (requestType.equalsIgnoreCase("pull") || requestType.startsWith("receipt") || requestType.equalsIgnoreCase("error")) {
                //correct parameters
            } else {
                throw new Exception("Method main needs first argument as: 'pull' 'receipt' or 'error' in case of 3 input parameters");
            }
        } else if (args.length == 4) {
            if (requestType.equalsIgnoreCase("push")) {
                String rawAttachInfo = args[3];
                String[] attachmentsRaw = rawAttachInfo.split(Pattern.quote("|"));

            for (String attachmentRaw : attachmentsRaw) {
                String[] attachmentRawCoupled = attachmentRaw.split(Pattern.quote("="));
                attachments.put(attachmentRawCoupled[0], attachmentRawCoupled[1]);
            }
            } else {
                throw new Exception("Method main needs first argument as: 'push' in case of 4 input parameters");
            }
        } else {
            throw new Exception("Method main needs 3 or 4 input parameters while it got: " + args.length);
        }

        String[] aliasKeyStorTruststoreInfo = args[1].split(Pattern.quote("|"));

        HashMap mapKeystores = new HashMap<>();
        mapKeystores.put("alias", aliasKeyStorTruststoreInfo[0]);
        mapKeystores.put("keystoreFile", aliasKeyStorTruststoreInfo[1]);
        mapKeystores.put("keystorePass", aliasKeyStorTruststoreInfo[2]);
        WSSSigner myMsgToSign = new WSSSigner(mapKeystores);

        String pathToUnsignedReqOrPullResponse = args[2];
        myMsgToSign.getSignedSoapMessageAsString(attachments, pathToUnsignedReqOrPullResponse);
        System.out.print(signedSoapMessage);  //this output is the clue of the application because SoapUI is grabbing it to its 'Request' field during Send action
    }
    
    //for troubleshooting
    public String helloWorld() throws Exception {
        //String sb = readFileFromClassLocation("/template_receipt.xml");
        //System.out.println("##################" + sb);
        return "Hello world from java jar file";
    }

    //used when some files are inside the JAR file
    // may be used for reading static templates that are not dependent on dynamic values
    public String readFileFromClassLocation(String path) throws Exception {
        String HoldsText;
        InputStream is = getClass().getResourceAsStream(path);
        InputStreamReader fr = new InputStreamReader(is);
        BufferedReader br = new BufferedReader(fr);

        StringBuilder sb = new StringBuilder();
        while((HoldsText = br.readLine())!= null){
            sb.append(HoldsText)
                    .append("\n");
        }
        return sb.toString();
    }

    public String getSignedSoapMessageAsString(HashMap attachments, String pathToUnsignedReqOrPullResponse) throws Exception {
        createSoapMessage(attachments, pathToUnsignedReqOrPullResponse);
        return signedSoapMessage;
    }
    
    public SOAPMessage createSoapMessage(HashMap attachments, String pathToUnsignedReqOrPullResponse) throws Exception {

        // Create message
        MessageFactory mf = MessageFactory.newInstance(SOAPConstants.SOAP_1_2_PROTOCOL);
        SOAPMessage msg = mf.createMessage();
        String fileContent = FileSystemMethods.readFile(pathToUnsignedReqOrPullResponse);

        //part for 'positive receipt' and 'error receipt'
        if (WSSSigner.requestType.equalsIgnoreCase("receipt")) {
            String template = readFileFromClassLocation("/template_receipt.xml");
            fileContent = XmlTransformation.preparePositiveReceipt(template, fileContent, "receipt");

        } else if (WSSSigner.requestType.equalsIgnoreCase("error")) {
            String template = readFileFromClassLocation("/template_error.xml");
            fileContent = XmlTransformation.prepareErrorReceipt(template, fileContent);
        }

        // Object for message parts
        SOAPPart soapPart = msg.getSOAPPart();
        InputStreamReader isr = new InputStreamReader(IOUtils.toInputStream(fileContent));
        StreamSource prepMsg = new StreamSource(isr);
        soapPart.setContent(prepMsg);
        
        for (Map.Entry entry : attachments.entrySet()) {
            String contentID = entry.getKey();
            String filePath = entry.getValue();
            
            //adding attachment part
            File f = new File(filePath);
            DataHandler dh = new DataHandler(new FileDataSource(f));
            AttachmentPart objAttachment = msg.createAttachmentPart(dh);
            objAttachment.setContentId("<" + contentID + ">");
            objAttachment.setContentType("application/gzip");
            objAttachment.setMimeHeader("Content-Transfer-Encoding", "binary");
            objAttachment.setMimeHeader("Content-Disposition", "attachment");

            msg.addAttachmentPart(objAttachment);
        }

        //saving message
        if(!requestType.equalsIgnoreCase("pull"))
            msg.saveChanges();

        logger.debug("Request SOAP Message:");
        msg.getSOAPPart().setTextContent(signSOAPEnvelope(msg));
        msg.saveChanges();
        return msg;
    }

    //most important method that is subject of this article
    //here the signing takes place
    public String signSOAPEnvelope(SOAPMessage unsignedMessage) throws Exception {
        
        SOAPEnvelope unsignedEnvelope = unsignedMessage.getSOAPPart().getEnvelope();
        Document doc = unsignedEnvelope.getOwnerDocument();

        WSSecSignature signer = new WSSecSignature();
        WSSecHeader secHeader = new WSSecHeader(doc);
        secHeader.setMustUnderstand(true);
        
        signer.setUserInfo(keystoreAlias, keystorePass);

        //parameters below depend on the algorithms being used by the server, so treat it as example
        //otherwise use different enums/values
        signer.setSignatureAlgorithm("http://www.w3.org/2001/04/xmldsig-more#rsa-sha256");
        signer.setSigCanonicalization(WSConstants.C14N_EXCL_OMIT_COMMENTS);
        signer.setDigestAlgo(WSConstants.SHA256);
        signer.setKeyIdentifierType(WSConstants.BST_DIRECT_REFERENCE);
        signer.setAddInclusivePrefixes(false);

        signer.appendBSTElementToHeader(secHeader); //appending Binary Security Token

        signer.setUseSingleCertificate(true);
        
        //signing specific parts, likewise in SoapUI
        signer.getParts().add(new WSEncryptionPart("Body", "http://www.w3.org/2003/05/soap-envelope","Element"));
        signer.getParts().add(new WSEncryptionPart("Messaging", "http://docs.oasis-open.org/ebxml-msg/ebms/v3.0/ns/core/200704/","Element")); 
                
        //create list of attachments to be signed, functionality that was lacking in SoapUI and all the fuzz was about
        Iterator iterator = unsignedMessage.getAttachments();
        List listAttachments = new ArrayList();
        boolean attachFiles = false;
        while(iterator.hasNext()) {
            attachFiles = true;
            AttachmentPart atttachmentPart = (AttachmentPart)iterator.next();
            Attachment attachment = new Attachment();
            attachment.addHeaders(getHeaders());
            attachment.setId(atttachmentPart.getContentId().replaceAll("<|>", ""));
            attachment.setSourceStream(atttachmentPart.getDataHandler().getDataSource().getInputStream());
            listAttachments.add(attachment);
        }
        
        if(attachFiles) {
            signer.getParts().add(new WSEncryptionPart("cid:Attachments", "Content"));
            AttachmentCallbackHandler attachmentCallbackHandler = new AttachmentCallbackHandler(listAttachments);
            signer.setAttachmentCallbackHandler(attachmentCallbackHandler);
        }
        
        secHeader.insertSecurityHeader();
        signer.prepare(doc, crypto, secHeader);

        Document signedDoc = signer.build(doc, crypto, secHeader);

        signedSoapMessage = org.apache.wss4j.common.util.XMLUtils.PrettyDocumentToString(signedDoc);
        //logger.info("Signed message: \n\r\n\r" + signedSoapMessage + "\n\r");
        
        return signedSoapMessage;
    }

    public static Map getHeaders() {
        Map headers = new HashMap<>();
        headers.put(AttachmentUtils.MIME_HEADER_CONTENT_DISPOSITION, "attachment");
        headers.put(AttachmentUtils.MIME_HEADER_CONTENT_TYPE, "application/gzip");
        headers.put("Content-Transfer-Encoding", "binary");
        return headers;
    }
}