gRPC Troubleshooting

-

Here are a few quick & dirty troubleshooting tips after a couple days in the trenches with a gRPC service (C# client, C++ server): 

  • Ensure any code creating a new alarm first cancels the last alarm it added to the queue, if it hasn't been processed.  In my case, I had a server-side observer receiving updates from elsewhere in the server module & writing them to the gRPC stream.  When those updates started coming too quickly, it crashed when adding the alarm to the queue.
  • Look for double disconnects.  I had another bug where the client was getting a cancellation exception from the server on the MoveNext() call, caught it, then proceeded to try to disconnect via Cancel() on the cancellation token.  The client then threw an exception: "Shutdown has already been called".
  • Check to see if your client is setting a deadline, which can cause it to close the connection.
  • Ensure your client code handles a cancellation exception coming from MoveNext().
  • Test by adding delays in various places.  One of my crashes was a race condition and would disappear with breakpoints at different points in the process.  Specifically, I think I had a handler trying to write to a stream after it was closed.  I would simultaneously get a cancellation exception on the client and have a new handler added to the completion queue.
  • Examine keep alives.  Services can be set up with keep alives sent periodically from the server.  Though, if you're going down under high load, this may not be your issue.
  • The common advice... Enable logging:
    • Environment.SetEnvironmentVariable("GRPC_TRACE", "all");
    • Environment.SetEnvironmentVariable("GRPC_VERBOSITY", "DEBUG");
    • I got this going on the client, but still haven't gotten it working on the server.  The client-side logs haven't been terribly helpful.

Comments

Post a Comment

Popular posts from this blog

Dude, Where's My Framework?

-
Image
You start out your day with a nice cup of coffee, and think, "Ah, greenfield project day...smooth sailing".  You fire up Visual Studio and create a new C# project.  "First things first, I need library X." you say.  "Wait, what the?"   The full error:  Package 'MyPackage 1.0.0.0' was restored using '.NETFramework,Version=v4.6.1, .NETFramework,Version=v4.6.2, .NETFramework,Version=v4.7, .NETFramework,Version=v4.7.1, .NETFramework,Version=v4.7.2, .NETFramework,Version=v4.8, .NETFramework,Version=v4.8.1' instead of the project target framework 'net6.0'. This package may not be fully compatible with your project.   "Ok" you think, "That library is a bit older.  I'll go update the library project to .NET 6 to match my project.  But, where is .NET 6?     "Ok, what about my new project?  Just as a test, does the warning go away if I set it to an older .NET Framework?  Wait, where are the .NET Framework versions?...
Read more

Permanently Mapping a Windows Share on Linux

-
Every few years, I have to set up a fresh Linux box, and I typically want to connect them to some Windows file shares.  There are many resources on how to do this, but many describe approaches which are incomplete, insecure, or not permanent (across reboots).  Here is what has worked for me. Note: The approach I describe comes from https://wiki.ubuntu.com/MountWindowsSharesPermanently.  I repeat it here because it gets buried in the search results (and I add a few details they gloss over & tweak a few things).  It is worth reading that as well.   Run these commands in order from the console.   Note: The two "nano" commands open a file editor. I have included the necessary file contents below. After updating/creating each of the files, save & exit the editor, and continue with these commands. You should now be able to browse to the local folders you entered in fstab and view the shares.
Read more

Initialize With Care

-
Pointers are funny things.  They are one of the make or break concepts for beginners, and even years later, they can cause grief to experienced developers.  I am no exception.  Here is one such story: I was faced with a class which I wanted to refactor.  It can be simplified as follows: In the interest of breaking up the responsibilities, I added a couple of interfaces. The idea is that I can pass around smart pointers to these interfaces and begin to decouple portions of the code.  For example, I can inject them into classes that need them: However, I made a mistake.  I blame it on years of using boost::intrustive_ptr instead of std::shared_ptr, but enough excuses.  Let's see if you can spot it. Do you see it?  If so, give yourself 5 points.  If not, maybe after seeing the output: 'second' going out of scope... Destructor 'first' going out of scope... 'root' going out of scope... All done... Both shared pointers ( first & second ) are ...
Read more