ENABLING RPC OVER HTTPS CONNECTIONS TO M-FILES SERVER

Transcription

1 M-FILES CORPORATION ENABLING RPC OVER HTTPS CONNECTIONS TO M-FILES SERVER VERSION 2.3 DECEMBER 18, 2015 Page 1 of 15

2 CONTENTS 1. Version history Overview System Requirements Network Layout Separate Proxy Server and M-Files Server Single Server Data security of "RPC over HTTP with SSL" Configuring the Servers Proxy Server M-Files server Configuring M-Files Desktop computers Disabling HTTPS Encryption Troubleshooting General troubleshooting Testing RPC Proxy settings Disabling compression Client trust for SSL certificate Copying configuration from a working computer Page 2 of 15

3 1. VERSION HISTORY Version Version notes 1.0 Initial version 2.0 Added additional information about data security (chapter 3.3). 2.1 SSL certificate instructions in chapter clarified 2.2 Recommendations in chapter clarified 2.3 Note about the proxy server running multiple sites added to chapter OVERVIEW By default, M-Files Desktop communicates with M-Files Server by using the Remote Procedure Call (RPC) protocol (TCP/IP, port 2266). This is typically the preferred means of communication within the company's internal network as it requires no additional configuration steps. In M-Files 10.2 and later, users can enable encryption for the RPC communication between M-Files Desktop and M-Files Server by turning on the "Encrypted connection" option in M-Files Desktop Settings (formerly M-Files Client Settings). In some situations, it is desirable to enable M-Files Desktop to communicate with M-Files Server via the HTTPS protocol instead of RPC. This is especially useful if clients are connecting from outside the company's internal network. HTTPS connections are always encrypted and are typically not blocked in hotel networks or other public networks. This document provides instructions for enabling "RPC over HTTP with SSL" communication between M-Files Desktop and M-Files Server. In this configuration, all traffic from M-Files Desktop is encrypted and tunneled through TCP port 443. See the document Protecting Data in Transit with Encryption in M-Files for information on choosing between "RPC Encryption" and "RPC over HTTP with SSL" SYSTEM REQUIREMENTS Client computers: Client computers must be running M-Files Desktop 10.0 Service Release 3 ( ) or later. Compatibility with older client versions requires additional configuration steps on the server. For instructions that are compatible with older M-Files Desktop versions, contact support@m-files.com. Server computer(s): Proxy server: - Windows Server 2003 Service Pack 2 or later. Windows Server 2008 R2 or later is recommended. M-Files server (application server): - Windows Server 2003 Service Pack 2 or later. Windows Server 2008 R2 or later is recommended. Page 3 of 15

4 - M-Files Server 9.0 or later. 3. NETWORK LAYOUT The HTTPS communication between M-Files Desktop and M-Files Server is based on the use of the RPC over HTTP protocol with SSL/TLS encryption. IIS (Internet Information Services) runs a component called RPC over HTTP Proxy that receives HTTPS traffic from the client and forwards it to M-Files Server as RPC calls. M-Files Server authenticates the user with the user's credentials. On the server side, there are two server roles in this setup: The server on which IIS is running is referred to as the "proxy server". The server on which the M-Files Server software is running is referred to as the "M-Files server". If you do not wish to separate IIS and M-Files Server to different servers, you can set up a single server that has both roles SEPARATE PROXY SERVER AND M-FILES SERVER The most common reason for separating the proxy server and M-Files server roles to separate servers is the additional security that this kind of isolation brings, especially if the proxy server is exposed to the public Internet. From the performance point of view, the roles do not typically need to be separated. Figure 1 shows the network layout with separate proxy server and M-Files server computers. Proxy Server in DMZ Public Internet User HTTPS (TCP port 443) DMZ Internal Network Firewall Software: - IIS with RPC over HTTP Proxy - M-Files Web Access (optional) RPC (TCP ports 2266 and 4466) RPC (TCP port 2266) Desktop user Proxy Server (dnsalias.domain.com) HTTPS (TCP port 443) M-Files Server (dnsalias.domain.local) User M-Files Server authenticates the user with the user s credentials. Figure 1 Network layout with a separate proxy server in DMZ If you set up a proxy server in addition to the actual M-Files Server as shown in Figure 1, you will typically have two separate DNS names that eventually lead to the same M-Files Server (e.g., "dnsalias.domain.com" and "dnsalias.domain.local"). To avoid confusion, you should use only one of those DNS names on any single client device. Page 4 of 15

5 For example, in M-Files Desktop, do not configure two connections that point to the same vault by using different DNS aliases SINGLE SERVER If the organization does not have a DMZ area in its network or does not want to set up a separate proxy server for other reasons, the role of the proxy server and the M-Files server can be combined (see Figure 2). No DMZ Public Internet User HTTPS (TCP port 443) Internal Network Firewall Software: - IIS with RPC over HTTP Proxy - M-Files Web Access (optional) - M-Files Server RPC (TCP port 2266) Desktop user M-Files Server (dnsalias.domain.com) HTTPS (TCP port 443) M-Files Server authenticates the user with the user s credentials. User Figure 2 Network layout without a separate proxy server Using a separate proxy server and placing it in the DMZ area of the network as described in section 3.1 provides additional isolation for M-Files Server and is the recommended approach DATA SECURITY OF "RPC OVER HTTP WITH SSL" When using RPC over HTTP with SSL, the HTTPS communication between M-Files Desktop and the proxy server is protected by SSL/TLS encryption. Encryption of this traffic is critically important because in many usage scenarios, this traffic travels over the Internet. On the proxy server, IIS runs a component called RPC over HTTP Proxy that receives HTTPS traffic from the client and forwards it to M-Files Server as RPC calls. The SSL/TLS encryption is decrypted on the IIS level. The RPC communication between IIS and M-Files Server is not encrypted. This portion of the traffic travels within an organization's server network and behind a firewall. The RPC over HTTP Proxy component in IIS can forward calls only to computers and ports that are specified in the ValidPorts registry entry on the proxy server. By specifying only the M-Files server computer and the port 4466 in the Page 5 of 15

6 ValidPorts entry, you can ensure that clients cannot attempt to communicate with any other RPC servers via the RPC over HTTP Proxy. In addition, before forwarding traffic to a given port on a target RPC server, the RPC over HTTP Proxy component performs a special packet exchange with the RPC server listening on that port to verify it is willing to accept requests over HTTP. RPC servers cannot accept RPC over HTTP calls unless they specifically requested RPC to listen on RPC over HTTP by specifying the "ncacn_http" protocol sequence. This behavior provides additional protection for RPC servers that listen on a port that is listed in the ValidPorts registry entry on the proxy server: unless the RPC server has specifically requested to listen on RPC over HTTP, it will not receive calls originating from outside the firewall. Optionally, you can strengthen the security of the M-Files system by enforcing the use of pre-shared key authentication on M-Files clients. See the documents Protecting Data in Transit with Encryption in M-Files and Securing Access to M-Files Vaults with a Pre-Shared Key for more information. Additionally, M-Files is compatible with SAML v2.0 compliant identity providers that support multi-factor authentication. See the document Deploying SAML v2.0 for M-Files Authentication for more information. 4. CONFIGURING THE SERVERS This section describes the needed configuration steps on the servers. The server on which IIS is running is referred to as the "proxy server". The server on which the M-Files Server software is running is referred to as the "M-Files server". If you do not wish to separate IIS and M-Files Server to different servers, you can set up a single server that has both roles. Follow the steps below to install and configure the needed software components on the servers PROXY SERVER The proxy server runs IIS. IIS receives the HTTPS traffic from the clients and converts it to RPC traffic for M-Files Server. Follow the steps below to install and configure IIS and the RPC over HTTP Proxy component on the proxy server INSTALL IIS AND RPC OVER HTTP PROXY Install IIS and RPC over HTTP Proxy on the proxy server: If the proxy server is running Windows Server 2003, follow these steps: 1. Choose Start / Control Panel / Add or Remove Programs / Add/Remove Windows Components. 2. In the Application Server group, turn on Internet Information Services (IIS). 3. In the Networking Services group, turn on RPC over HTTP Proxy. If the proxy server is running Windows Server 2008 or later, follow these steps: 1. In Server Manager, under Roles, choose Add Roles and add the Web Server (IIS) server role if not yet present. 2. In Server Manager, under Features, choose Add Features, and add the RPC over HTTP Proxy feature. Accept the adding of required role services if prompted. Finally, verify that Default Web Site exists, uses port 80, and is running. Page 6 of 15

7 Note: If the proxy server is running multiple sites, the RPC over HTTP Proxy feature must be added for the M-Files Web site. If M-Files Web is not the default web site of the proxy server, the following registry key needs to be added before installing the RPC over HTTP Proxy feature: Key name: Value name: Value type: Value data: HKEY_LOCAL_MACHINE\Software\Microsoft\Rpc\RpcProxy WebSite REG_SZ <The name of your M-Files Web site> CONFIGURE THE RPC PROXY After installing IIS and RPC over HTTP Proxy, configure the following registry values on the proxy server: Key name: HKEY_LOCAL_MACHINE\Software\Microsoft\Rpc\RpcProxy Value name: AllowAnonymous Value type: REG_DWORD Value data: 1 Key name: Value name: Value type: Value data: Remarks: HKEY_LOCAL_MACHINE\Software\Microsoft\Rpc\RpcProxy ValidPorts REG_SZ dnsalias.domain.com:4466 If you use RPC over HTTP Proxy only for the purpose of enabling RPC over HTTPS connections to M-Files Server, you can replace any existing ValidPorts string value (e.g. servername: ) with dnsalias.domain.com:4466. This provides maximum security because RPC over HTTP Proxy will not forward RPC calls to any other computer or port. If you use RPC over HTTP Proxy for other, non-m-files purposes as well, you should append ;dnsalias.domain.com:4466 at the end of the other ValidPorts settings that you need. Above, dnsalias.domain.com is the fully qualified DNS name that the clients use to connect to M-Files Server. This is typically the DNS name of the proxy server or a separately created DNS alias such as "mf.domain.com" is the default port used by M-Files Server for receiving RPC calls from IIS. The purpose of the ValidPorts entry is to enable IIS to forward the received traffic as RPC calls to M-Files Server to port IIS will only forward traffic to targets for which an exactly matching server name and port number are found in the ValidPorts entry. If the ValidPorts entry is missing or incorrect, the traffic will stop at IIS. Note: When the proxy server and M-Files server are separate servers, you must create a new DNS alias such as "mf.domain.com" and configure it to point to the proxy server's IP address in the DNS system. On the proxy server, this DNS name will then be mapped to the M-Files server's IP address by using a HOSTS file entry (see below). Do not use the proxy server's own DNS name. Using a separate DNS alias is required because the Windows Server operating system may refuse to read an IP address override from the HOSTS file for the server's own DNS name MODIFY THE HOSTS FILE In the proxy server's HOSTS file in C:\Windows\System32\drivers\etc, map the fully qualified DNS name that the clients use to connect to M-Files Server to the IP address of the M-Files server. IIS will forward the RPC calls to M-Files Server based on the IP address specified in this entry. If the M-Files server is the same computer as the proxy server, use as the IP address. For example: Page 7 of 15

8 dnsalias.domain.com # Map the DNS name that M-Files clients use to the IP address of M-Files Server If the M-Files server is a separate server, use the IP address of the M-Files server. For example: dnsalias.domain.com # Map the DNS name that M-Files clients use to the IP address of M-Files Server Note: IIS will forward the RPC calls to the same DNS name that the clients used for connecting to the proxy server. Thus, the DNS name that the clients use in their vault connection settings must resolve to the IP address of the proxy server on the client computers. However, on the proxy server itself, this DNS name must resolve to the IP address of the M-Files server. This is achieved by mapping the DNS name to the proxy server's IP address on the DNS servers, and overriding this on the proxy server by using the HOSTS file to map the same DNS name to the M-Files server's IP address instead. For example, in the scenario that is shown in Figure 1, the entry added to the HOSTS file on the proxy server should have the DNS name of the proxy server (dnsalias.domain.com) but the IP address of the M-Files server (i.e., the IP address of dnsalias.domain.local, NOT the IP address of dnsalias.domain.com) INSTALL AN SSL CERTIFICATE Get and install an SSL certificate for the Default Web Site in IIS on the proxy server. Remember to add an HTTPS binding to the website in IIS and configure it to use the SSL certificate you acquired. You can get the certificate at for example. An SSL certificate is required for encrypted HTTPS traffic. The client computers must trust the SSL certificate of the server. If you use an official SSL certificate this is true by default. Note: Always use an official SSL certificate issued by a trusted Certification Authority (CA). Using a self-issued SSL certificate is not recommended because it adds a lot of complexity to the configuration of client computers. You would need to ensure that all client computers trust the self-issued SSL certificate both in the user's context and in the Local System account's context in which the M-Files Desktop service is running. If any part of the certificate trust configuration is incorrect, clients will fail to connect and will only report a generic network error. For this reason, using an official SSL certificate from e.g. is highly recommended. If you use a self-issued SSL certificate, you must install the issuing CA's Root CA Certificate on each client computer. The root certificate must be available both in the user's context as well as in the computer account's context. See the following article on how to configure Computer Account level certificates: CONFIGURE THE RPC VIRTUAL DIRECTORY In IIS on the proxy server, inside Default Web Site, configure the Rpc virtual directory's Authentication settings: a. Allow Anonymous Authentication (typically disabled by default). b. Allow Basic Authentication (typically already enabled) ADD HANDLER MAPPING (IIS 7 ONLY) If the proxy server is running IIS version 7, it may be necessary to manually create a handler mapping for the RPC Proxy DLL. Without this, IIS 7 may block requests to the DLL and will fail. Adding this setting is recommended only once you have first completed all other steps and determined that the connection is not working properly. Page 8 of 15

9 To add a handler mapping manually, select Default Web Site, go to Handler Mappings, click the Add Script Map link on the right and enter the following values: Figure 3 Edit Script Map in IIS CONFIGURE FIREWALL In the proxy server's firewall, allow incoming traffic to TCP port 443 (HTTPS port) to enable clients to communicate with the server computer via HTTPS. You can disable all other incoming traffic (except typically Remote Desktop, which is used for managing the server). You can also disable plain HTTP (TCP port 80) RESTART IIS On the proxy server, restart IIS for the above settings to take effect. Recommended: Disable SSL 2.0, SSL 3.0 and RC4 on the proxy server to improve security. Disabling SSL 2.0 and SSL 3.0 ensures that clients can only connect using TLS 1.0 or newer. Disabling RC4 ciphers is recommended to avoid using unsecure ciphers during the encryption. To disable SSL 2.0, SSL 3.0 and RC4, add the registry values of the attached file Disable SSLv2 and SSLv3.reg.txt on the proxy server and restart the server computer. Note: Restarting IIS is not sufficient for the disabling of SSL 2.0, SSL 3.0 and RC4 to become effective. A server restart is required for these settings to take effect M-FILES SERVER The M-Files server computer runs the M-Files Server software. M-Files Server receives RPC calls from IIS on the proxy server. This document assumes that the M-Files Server software has already been installed on the M-Files server. Follow the steps below to configure M-Files Server to accept RPC over HTTP traffic. Page 9 of 15

10 4.2.1 ENABLE RPC OVER HTTP TRAFFIC On the M-Files server, add the following registry value to enable M-Files Server to accept RPC over HTTP traffic: Key name: HKEY_LOCAL_MACHINE\Software\Motive\M-Files\<version>\Server\MFServer Value name: EnableRPCOverHTTP Value type: REG_DWORD Value data: 1 Note: Verify that the value name does not contain a trailing space CONFIGURE FIREWALL In the M-Files server's firewall, allow incoming traffic to TCP port 4466 to enable M-Files Server to receive RPC calls from IIS on the proxy server. Additionally, you should allow incoming traffic to TCP port 2266 in the M-Files server's firewall if M-Files Desktop users may be connecting to M-Files Server also using the default RPC protocol (TCP/IP, port 2266) or if the proxy server is also running M-Files Web Access. Communication between M-Files Web Access and M-Files Server uses the default RPC protocol (TCP/IP, port 2266) if M-Files Web Access is running on a separate server RESTART M-FILES SERVER Restart the M-Files Server service for the above settings to take effect. 5. CONFIGURING M-FILES DESKTOP COMPUTERS Open M-Files Desktop Settings and add or edit a document vault connection. Select HTTPS as the protocol and set port number to NOTE: The port number 4466 is used only for traffic between the proxy server and the M-Files server computer. All traffic from M-Files Desktop to the proxy server is encrypted and tunneled via TCP port 443 (HTTPS). On the proxy server, only the port 443 (HTTPS) needs to be opened in the firewall. See Figure 4 for sample settings for the document vault connection in M-Files Desktop Settings: Page 10 of 15

11 Figure 4 Sample settings in the M-Files Desktop Settings tool 5.1. DISABLING HTTPS ENCRYPTION In some cases it is desirable to disable the use of SSL/TLS encryption between M-Files Desktop and the proxy server and instead use RPC over HTTP without encryption, i.e., to use plain HTTP instead of HTTPS. Without SSL/TLS encryption, the traffic from M-Files Desktop towards the proxy server will be directed to the standard HTTP port (TCP port 80). You should disable the use of SSL/TLS encryption only if you have secured the communication by some other means. To disable SSL/TLS encryption, add the following registry value on all client computers: Key name: HKEY_LOCAL_MACHINE\Software\Motive\M-Files Value name: EnableSSL Value type: REG_DWORD Value data: 0 (default = 1) After changing the setting, restart the M-Files Desktop service. On the server, verify that IIS settings for Default Web Site as well as for the Rpc virtual directory allow non-ssl traffic and that the default http binding (port 80) is enabled. Also ensure that the firewall on the proxy server allows incoming traffic to TCP port 80. When SSL/TLS encryption is not used, you do not need to install an SSL certificate in IIS on the proxy server. Page 11 of 15

12 6. TROUBLESHOOTING 6.1. GENERAL TROUBLESHOOTING The configuration of RPC over HTTP is fairly complex, and if any part of the configuration is incorrect, the connection from the client computer to the server is likely to fail. If the connection is not working and the error message does not immediately reveal the cause of the problem, you should first review all the settings carefully and double-check that the settings have been configured properly on the proxy server and the M-Files server as applicable. Error messages that refer to "network problems preventing communication with the server" indicate that something is preventing the communication from reaching the M-Files Server service. Potential causes include: 1. Firewall not allowing incoming HTTPS traffic on the proxy server. 2. IIS not running on the proxy server. 3. Default Web Site not started in IIS on the proxy server. 4. Default Web Site not configured for the default bindings in IIS on the proxy server (https, port 443). 5. SSL certificate not installed in IIS on the proxy server. 6. SSL certificate expired or otherwise not valid. 7. SSL certificate not issued by a trusted Certification Authority (CA). 8. Rpc and/or RpcWithCert virtual directories missing from IIS on the proxy server (RPC over HTTP Proxy not properly installed). 9. ValidPorts registry value not properly configured on the proxy server or refers to a different DNS name or port number than what the clients use for connecting to the server. 10. HOSTS file entry not added on the proxy server. 11. HOSTS file entry referring to a different DNS name than what the clients use for connecting to the server. 12. HOSTS file entry mapping to a different IP address than the IP address of the computer that is running the M-Files Server software. 13. IIS has not been restarted after changing the settings. 14. Firewall not allowing incoming TCP traffic to port 4466 on the M-Files server. 15. EnableRPCOverHTTP registry value not properly configured on the M-Files server. 16. M-Files Server has not been restarted after changing the settings. 17. Spelling errors in registry settings or trailing spaces in registry setting names, e.g., "EnableRPCOverHTTPS" with an extra S or "EnableRPCOverHTTP " with an extra trailing space instead of the correct spelling "EnableRPCOverHTTP". You can export the registry branch and review the exported REG file's content to make it easier to notice especially any trailing spaces. An "Access denied" error message may indicate that some of the authentication-related settings are not correct. Potential causes include: 1. Anonymous Authentication not enabled for the Rpc virtual directory under Default Web Site in IIS on the proxy server. 2. The AllowAnonymous registry value not properly configured on the proxy server. 3. IIS has not been restarted after changing the settings. The following sections describe error conditions that may be encountered under special circumstances. Page 12 of 15

13 6.2. TESTING RPC PROXY SETTINGS You can test the RPC Proxy by pointing your Web browser to the URL The "proxyserverurl" is the fully qualified domain name (FQDN) of the proxy server, e.g., "dnsalias.domain.com". The page should ask for credentials. Enter Windows credentials which have permission to log on to Windows on the proxy server and press OK. A blank page as a result means that the RPC Proxy appears to be working OK. If instead of a blank page you receive an HTTP error page, this may indicate that the RPC Proxy is not correctly configured. Some known errors are listed below. If there are other errors (e.g. plain "Access Denied" text on the page), the test is inconclusive and you should check for other possible error cases. It may be necessary to open the URL by using a Web browser on the proxy server itself to get the detailed error code. However, this can work properly only if the proxy server and the M-Files server are the same computer. If they are separate servers, then opening the above mentioned URL locally on the proxy server will fail because the HOSTS file entry causes the DNS alias to be mapped to the IP address of the M-Files server, which does not have the RPC Proxy. This can be worked around by temporarily modifying the HOSTS file entry to map the DNS alias to After the test, the HOSTS file entry must be restored to map the DNS alias to the IP address of the M-Files server. Some known errors are listed below ERROR UNAUTHORIZED This error appears at least when you press Cancel in the credentials dialog. Try closing the browser, and then entering valid Windows credentials to the dialog ERROR : INTERNAL SERVER ERROR Make sure that DefaultAppPool -> Advanced Settings -> Enable 32-bit Applications is False. Restart IIS ERROR 404: NOT FOUND Make sure that RPC Proxy Server Extension ("%windir%\system32\rpcproxy\rpcproxy.dll") is allowed in the ISAPI and CGI restrictions configuration in IIS settings ERROR 405: METHOD NOT ALLOWED Check Handler Mappings: ISAPI-dll should be enabled. If it isn't, select it, edit feature permissions, and add Execute ERROR 500.0: INTERNAL SERVER ERROR Error Code: 0x f There is a problem with the resource you are looking for, so it cannot be displayed. This may occur if the default website has been deleted and Rpc has been manually added to another site. By default, IIS tries to look up Rpc from the default website instance (with ID 1). Page 13 of 15

14 To solve the problem, add the following registry value on the proxy server: Key name: Value name: Value type: Value data: HKEY_LOCAL_MACHINE\Software\Microsoft\Rpc\RpcProxy Website REG_SZ Website name (e.g. "Default Web Site") Additional resolution suggestions: REQUESTS HANG UNTIL RESTARTING IIS Seems to be caused by "<serverruntime uploadreadaheadsize="0" />" missing from C:\Windows\System32\inetsrv\config\ApplicationHost.config under <location path="default Web Site/Rpc"> <system.webserver>. Add the value and restart IIS DISABLING COMPRESSION Some sources like suggest disabling compression may solve some problems, at least HTTP error which references DynamicCompressionModule. Disable the compression with the following command-line parameters: %windir%\system32\inetsrv\appcmd.exe list config -section:system.webserver/httpcompression %windir%\system32\inetsrv\appcmd.exe set config -section:system.webserver/httpcompression /-[name='xpress'] To re-enable compression: %windir%\system32\inetsrv\appcmd.exe set config -section:system.webserver/httpcompression /+[name='xpress',dostaticcompression='false',dll='%windir%\system32\inetsrv\suscomp.dll'] 6.4. CLIENT TRUST FOR SSL CERTIFICATE Because the M-Files Desktop software uses the Local System account, the trust for the SSL certificate has to be defined for the Computer Account on the client computer. If the SSL certificate trust is OK on the user account level but missing on the Computer Account level, testing the connection from the M-Files Desktop Settings tool succeeds but navigating to the virtual M-Files drive in Windows Explorer displays a generic "network problems preventing communication with the server" error. See the following page on how to configure Computer Account level certificates: COPYING CONFIGURATION FROM A WORKING COMPUTER If the fixes suggested above do not help resolve the issue, you can try to copy a configuration from a working IIS web server as suggested in Page 14 of 15

15 Open C:\Windows\System32\inetsrv\config\ApplicationHost.config from a working installation. A sample file is also attached to this documentation. Take a backup of the same configuration file in the problematic installation. Replace the problematic installation's <application> and <location> tag contents with ones from the working configuration. Restart IIS. Page 15 of 15