SFTP

SFTP (Secure Shell File Transfer Protocol) serves as a critical component in integration environments, facilitating secure and reliable file transfer between systems.

Let's explore a real-time situation to understand the importance of SFTP monitoring: Imagine an integration environment where the SFTP Adapter ensures the timely processing of orders and invoices by securely transferring data from external sources. In this scenario, handling orders and invoices must process the data by noon to meet delivery deadlines, Otherwise, end users might face a delay in delivery, affecting the company's revenue. What if any of the SFTP pickups didn't process the orders for a long period, causing them to be stuck in the location? Orders not processed before the deadline result in lost money! Manually checking the numerous SFTP ports periodically isn't feasible, and issues may arise on any port and at any time, sometimes even in the midnight. This underscores the critical role of SFTP monitoring in ensuring seamless data exchange.

Using BizTalk360, users can regularly monitor SFTP locations for file count, directory size, and message age to ensure smooth data flow and promptly identify processing delays, notifying relevant stakeholders.

Setting up SFTP Monitoring

You can follow the below steps to configure SFTP monitoring:

• Log in to the BizTalk360 application.
• Go to Monitoring > Manage Mapping > File Locations section. 
• You can see three tabs each containing port details of their respective adapters, which are File, FTP, and SFTP.
• Click the 'SFTP' tab to monitor SFTP Locations.
• Select the Alarm (Manage Alarms) from the drop-down, which you would like to associate the SFTP location for monitoring.
• The receive locations and send ports that use the "BizTalk SFTP" and "nsoftware SFTP" adapters are displayed in a grid. 
• Click on the SFTP location to configure the details and threshold rules for monitoring.

The SFTP details blade contains the following sections:

1. SFTP Configuration
2. Security Details
3. Monitoring Rules
4. Restart Host Instance (Applicable only for Receive Locations)

All fields in the SFTP Configuration and Security Details are editable. By default, details are pre-populated from configurations in the BizTalk Admin Console. Users can overwrite these configurations if required.

1. SFTP Configurations

• The Folder Location, Server, File Mask, Port, Connection Timeout and Proxy Settings that will be used for monitored is displayed here.
• File Mask: If the folder might receive multiple types of files, but user wants to monitor only a particular type of files, they can overwrite the File Mask. All file extensions (e.g., CHPAPO*.txt, *.txt, *. *, *outbound322*, PPI*, Test*. *) and macros patterns are supported.
• Adjust the Connection Timeout settings between 5 to 300 seconds, specifying the maximum time allowed for connecting with SFTP locations.
• Provide the Proxy Settings if it is required for connection. If you want to use the default proxy configuration given in "Settings > Gateway and Proxy settings", enable the 'Use Gateway Proxy' option.

Monitoring Absolute Path vs Relative Path

In SFTP ports, you can specify folder locations either as absolute paths or relative paths. BizTalk360 supports both Absolute Path and Relative Path for the folder location, making it even more versatile and useful for users. Let's explore how each option works in BizTalk360.

Absolute Path: 

An absolute path refers to the precise location of a file or directory starting from the root directory. In simpler terms, it represents the complete path starting from the root directory (E.g. "C:") of the filesystem. When an absolute path is used, BizTalk360 recognizes it as a fixed folder location on the SFTP Server.

For example,  if the folder location is specified as  "C:/BizTalk/IncomingOrders", BizTalk360 will check the exact path specified.

Relative Path:

A relative path specifies the location of a file or directory in relation to the current working directory. Current working directory is nothing but the default path configured for the user in sshd configuration.

When monitoring with a relative path, the folder location specified is interpreted relative to the default path.

For example, if the default path is set as "C:/Orders" in the sshd configuration, and the folder location is specified as "../Customers/Euro/In" in BizTalk360, then the final full path that will be monitored is "C:/Customers/Euro/In". 

"../ in the file path indicates to navigate one level up in the directory structure.

"./ in the file path indicates the current directory.

2. Security Details

• Turn on the “Accept Any SSH server host key” toggle button if you don’t want to provide the “SSH Server Host Key Finger Print”. If it is turned off, then “SSH Server Host Key Finger Print” needs to be provided.
• The following four types of Request Methods are supported in BizTalk360.
  1. Password - This is the simplest form of authentication using the traditional username/password method. Enter the Username and Password to be used for authentication.
  2. Public Key Authentication - Public key authentication is a method where the SFTP client identifies itself to the server by using Public / Private Key pairs. Enter the Username, Private Key and Private Key Password to be used for authentication. 
  3. Multi-Factor Authentication - For this both password and public key authentications are performed. Enter the Username, Password, Private Key and Private Key Password to be used for authentication. 
  4. Keyboard Interactive Authentication - Enter the Username and Password to be used for authentication. 

3. Monitoring Rules

• Users can configure the Warning and Error threshold rules for different metrics (File Count, Directory Size and Age) that they want to monitor. 
• Here is a combination of rules, that helps to identify the issue for the latter mentioned scenario. In this case, if any order stays in the SFTP location for more than one hour, it will trigger an alert to respective users. 
  Furthermore, it will also automatically restart the associated Host Instance on detecting the violations, that will act as an immediate remedy to process the orders further.
• Once after saving the details and rules, if BizTalk360 can connect with the SFTP location you can view the current File Count, Age and Directory Size of the location.
• Click on "eye" icon at the end of the row to view the violated rule.

When and Why SFTP Locations go Orphaned?

If BizTalk360 cannot connect to the SFTP location, the configuration will be considered Orphaned and monitoring the metrics will not be carried out. 

The error message indicating the orphaned reason will be displayed near the port name. User can view the error code by hovering over the warning icon near the Port name.

Possible reasons for Orphaned Locations:

• Any of the provided Configuration or Security details are invalid - Ensure the provided details are valid and correct.
• The SFTP Server is not reachable at the moment - Check if the SFTP server is reachable and allowed for remote connection using the provided authentication details.
• A timeout occurred while attempting to connect with the server - Increase the 'Connection Timeout' in BizTalk360.
• The configured SFTP location is removed or renamed in the machine but not updated in BizTalk360 configuration - Update the BizTalk360 Configuration with latest values (or) remove and re-configure the location.
• When the respective Application or Port is modified in the BizTalk Admin Console but not updated in BizTalk360 configuration - Remove and re-configure the location 
• BizTalk360 cannot find the folder at the specified location - Ensure the folder is available at the specified path.