OK so sometimes just posting a question helps me to think through a solution. Here is what I was able to come up with overnight. I was considering the formula from a tilt compensation paper I have been referencing where they describe the basic heading formula applied to the magnetometer x/y values such that heading=atan(Hy/Hx). This got me thinking that applying the inverse of that function back to the angle would give me the ratio of the x/y values. I would then only need multiply an arbitrary x value to get a y value I could work with in the remainder of the tilt compensation equations from the article.
Here as an example I have worked out: Lets say my compass is reading 48 degrees to start. For all of my calculations I have decided 20 will be my x value. This value is arbitrary, as it is only being used as a multiplier to the x/y ratio to find the y value. So, I take the tangent of my angle 48, which is about 1.11. That is the ratio of my x/y values. Now I multiply that by my arbitrary x value of 20, so 20*1.11 gives me 22.2, which should be close to my y value. To check, I just plug x=20, y=22.2 back into my original heading formula heading=atan(Hy/Hx), which is heading=atan(22.2/20), which is heading=atan(1.11), which is heading=47.984. So far so good!
Thus far I have only worked this out in the 0-90 degree quadrant. The example formulas in the text I am referring to work out the quadrants later in a series of steps in order to return the appropriate 0-360 degree returns after tilt compensation. I will post back if and when I get the rest worked out. This may seem like a roundabout approach to the problem, but now that I have worked through it it is really only a few steps and not that much to add to the overall process. Perhaps this might help someone else working on the same problem. It also illustrates that a packaged magnetic compass might be convenient if all one is looking for is magnetic heading, but if one wishes to do further corrections from the x/y/z magnetometer readings, they might be better off just starting with a 3-axis magnetometer and using the raw values.
Here is the link to the paper I referenced. It is about as thorough an explanation of magnetic compass tilt compensation as I have ever found:
https://www.cypress.com/file/130456/download
Perhaps it might help others working with the same problem.- Rob