Demystifying the rpcinfo and rpcbind Commands in Linux: An In-Depth Reference

As a seasoned Linux engineer well-versed in distributed systems, I often rely on the rpcinfo and rpcbind commands while configuring or debugging remote procedure call (RPC) services. RPCs act as the glue for various critical network functions – NFS mounts, NIS user auth, DNS, and more. So fully grasping these RPC tools is key for any aspiring or expert Linux admin.

In this comprehensive reference, we’ll unpack the internals of RPC communication, analyze rpcbind and rpcinfo usage via examples, and cover best practices for securing RPC. Buckle up for a deep dive into the RPC toolchain!

Inside Remote Procedure Calls

To understand rpcbind and rpcinfo, we first need to explore how RPCs work under the hood. RPCs facilitate client/server interactions between network-connected processes. Clients call procedures as if locally, yet executed on remote hosts.

Here’s an high-level overview:

1. A server defines RPC procedures and registers them with the rpcbind daemon, including the program number and port.

2. The client loads required stubs to call remote procedures.

3. The client stub contacts rpcbind on the server‘s host to lookup the program‘s address.

4. rpcbind replies with the server‘s binding details.

5. The client calls the remote procedure via its stub. Parameters and return values are marshalled across the network.

6. The server stub receives the call, executes procedures locally, and returns output which is sent back to the client transparently.

This demonstrates how rpcbind acts as the directory facilitating RPC communications by tracking available registered services. The client and server stubs handle marshalling of data across process and host boundaries.

Now let‘s analyze some key mechanisms and formats in an RPC call:

Network Address & Port Lookup

At its core, rpcbind maps RPC service names to network addresses and ports. This enables dynamically looking up the endpoint for an RPC program‘s procedures.

The lookup query specifies the RPC program number assigned when it was compiled via rpcgen. rpcbind‘s response provides thetransport protocol (TCP/UDP), IP address, and port.

For instance, an NFS program number query would retrieve udp/tcp port 2049 on the NFS server.

XDR Data Serialization

Parameter passing between RPC client/server requires formatting and de-formatting data across architectures and operating systems. This is handled by eXternal Data Representation (XDR) – a standard for encoding data types like strings, integers, booleans, structures, etc.

XDR outlines rules for serializing application data onto an architecture-agnostic wire format, then reconstructing the data into applicable native formats. This allows RPC to facilitate communication between different hardware and operating systems.

Authentication

While RPC communication may be unauthenticated, common standards like RPCSEC GSS utilize kerberos for security:

• Associating credentials to verify the remote user‘s identity
• Encrypting parameters to provide confidentiality
• Signing messages to ensure integrity checked

Securing RPC is vital for production services, explored later below.

Diving Into rpcinfo Usage

Now that we‘ve established the RPC foundations, let‘s analyze rpcinfo functionality and usage in-depth using demonstration examples.

The rpcinfo command interrogates remote RPC daemons to report active registered services and statistics. Common uses include:

1. Checking availability of key RPC services
2. Testing connectivity to RPC endpoints
3. Profiling statistics to analyze performance

Let‘s explore each of these activities in detail:

Checking RPC Services Status

A common task is verifying availability of important RPC services across our environment. For example, we may need to check NFS status on multiple storage servers.

Use the -p probe parameter to dumps all services registered locally via rpcbind:

$ rpcinfo -p storage01

  program version(s) netid(s)                service    owner
    100000    4,3,2    tcp,udp,tcp6,udp6  rpcbind    superuser
    100005    1,2,3    tcp,udp,tcp6,udp6  mountd     superuser
    100003    2,3,4    tcp,udp,tcp6,udp6  nfs        superuser
    100011    1        udp,tcp,udp6,tcp6  rquotad    superuser
    100024    1        udp,tcp,udp6,tcp6  status     superuser
    100021    1,3,4    tcp,udp,tcp6,udp6  nlockmgr   superuser
    100003    2,3,4    tcp,udp,tcp6,udp6   nfs4      superuser

Great – NFSv3 and NFSv4 are registered along with supporting daemons like mountd and nlockmgr. This storage server is ready to handle NFS RPC requests.

Sometimes though a service registration disappears or rpcbind dies. So monitoring -p output is key to catch issues quick.

Testing RPC Connectivity

Verifying we can establish RPC connections for a service session is also critical. The -t test parameter handles this by sending a NULL procedure call then timing the delay waiting for a response.

For example, testing NFS connectivity on storage02‘s default NFS TCP port:

$ rpcinfo -t 192.168.2.100 2049 nfs
  program 100003 version 2 ready and waiting
  program 100003 version 3 ready and waiting
  program 100003 version 4 ready and waiting

Great – we successfully contacted the NFS service! The ready and waiting status indicates outstanding readiness to handle our NFS traffic.

Contrast that working connection with a failed one:

$ rpcinfo -t 192.168.2.100 2049 nfs
rpcinfo: RPC: Program not registered

Troubleshooting this would require confirming NFS is still running on that server and registered properly with its rpcbind daemon. So -t serves as a quick first connectivity check.

Statistics and Performance

Examining rpcinfo statistics provides valuable performance insights into how active and responsive an RPC service is operating.

The -s parameter retrieving call stats for defined programs like NFS or NIS. For example, checking v4 NFS stats on storage02:

$ rpcinfo -s 192.168.2.100 100003 4 

Program: 100003 (nfs), version 4
  rcdir_nocreate (NULL)           calls      106         6.3% 0.0 min       92.0 avg     92.0 max 
  commit                           calls        7         0.4% 0.0 min        0.4 avg      1.0 max
  write                            calls     2558       153.7% 0.0 min        2.0 avg     32.0 max
  directory_remove                 calls       14         0.8% 0.0 min        5.1 avg     20.0 max

This shows active metrics for every NFSv4 procedure:

• calls: Number of times procedure called
• %: Percentage of total calls
• min/avg/max: Minimum, average and max milliseconds per call

Compare loads between versions with separate queries:

$ rpcinfo -s 192.168.2.100 100003 3
Program 100003 version 3:
  null                             calls    0
  getattr                          calls    9887
  setattr                          calls    237
  ...

We can also summarize rates for all procedures with -a:

$ rpcinfo -a 192.168.2.100 100003
  program 100003 nfs v2,3,4
  rcdir_nocreate null start 0 1598749394 (2443s)
  commit        null start 0 1598749394 (2443s) 
  write         null start 0 1598749394 (2443s)
...

Monitoring trends in stats or sudden surges in rates helps diagnose remote procedure issues hampering distributed application performance.

So in summary, rpcinfo reporting unravels the mystery whether slowness stems from the app logic itself or the communication foundation.

Securing Remote Procedure Calls

While enabling seamless interactions between distributed programs, RPC communication opens potential attack surfaces. Call parameters could contain sensitive customer data or authentication tokens. Plus RPC exploits can lead deeper into the application architecture.

Given RPC underpins vital system functions like DNS and Kerberos, we must implement protections:

Enable RPC Authentication

As mentioned earlier, RPCSEC GSS allows applying kerberos infrastructure to verify identities and encrypt transport:

 rpcbind -a svc_auth.gss

Additionally make use of TLS certificates or SSH tunnels to encrypt from endpoint to endpoint across the wire.

Isolate Services With Firewalls

Surrounding rpcbind with a tightly controlled firewall limits connectivity to other hosts. Bind specifically to internal subnets instead of 0.0.0.0.

Also firewall the application ports mapped by rpcbind like NFS or mail services.

Monitor Authorization Logs

Centralized logging coupled with log monitoring tools like Splunk allows watching for suspicious RPC patterns – floods of calls generating access denied events for example.

Keep Software Up to Date

Given RPC‘s breadth across Linux system services, staying current with patch management is crucial to incorporate the latest security enhancements.

Alternatives to Remote Procedure Calls

While RPCs solve the need for communication between distributed programs written across languages, operating systems, and architectures – modern replacements provide greater speed, reliability, and security.

For new applications, engineers may choose:

• HTTP-based REST or GraphQL APIs: Leveraging stateless web architecture

• Message queues: Async loosely-coupled event processing

• gRPC: Google‘s high-performance RPC framework

• Thrift: Apache cross-language RPC stack

These alternatives overcome technical shortcomings depending on use cases. But replacing major RPC-dependant Linux services like NFS or Kerberos represents significant porting effort.

So RPC powers on as a legacy protocol – yet with rpcbind and rpcinfo, admins can still manage these workhorses effectively.

Conclusion

This exhaustive reference aimed to solidify understanding of RPC communication flows, dig into rpcinfo analysis, and provide security best practices. Key takeaways include:

• rpcbind tracks registered RPC program addresses required for clients establishing connections
• rpcinfo allows reporting on RPC service availability as well as performance statistics
• Common use cases involve testing connectivity, debugging issues, and profiling workloads
• Properly configuring RPC authentication, isolation, and monitoring is essential

The rpcinfo and rpcbind commands will equip you to orchestrate the pillar RPC infrastructure powering essential functions in the enterprise data center. Master these tools to enhance distributed application proficiency during your journey to advanced Linux engineer!