SharePoint patches coming to Windows Update

Word of warning: If you weren’t already convinced to turn automatic Windows updating off on your SharePoint servers, here’s a little more food for thought…

http://blogs.technet.com/b/stefan_gossner/archive/2015/02/12/sharepoint-cus-and-windows-update.aspx 

With the February 2015 cumulative update, Microsoft will be rolling CUs into Windows Update. So, if your server is set to automatically download and install Windows Updates, you’re going to be getting the latest and greatest CUs every time.

Uh oh.

You don’t have to have the memory of an elephant to know Microsoft has a spotty history with their CUs, PUs, and even Service Packs for SharePoint Server. And, of course, recent issues with desktop patching also fail to reassure

At TCSC, we advise our clients to stay two Cumulative Updates behind the current CU when installing a new environment (unless there’s a specific reason to implement a particular CU), and not to concern themselves with keeping up with CUs overmuch after go-live unless a specific CU will resolve a specific issue they are experiencing*. If a SharePoint administrator wants to keep to this or a similar recommendation, the only way now is to turn off automatic Windows Updates.

Is Microsoft sneakily trying to make the SharePoint Server story more frustrating in order to drive more customers to the cloud? Well, I wouldn’t exactly say that… but, hey, by the way, if you do find administering SharePoint Server frustrating, SharePoint Online might be a good option for you…  :)

*Public Updates, or PUs, and Service Packs should be installed when released, though with caution. Unless it’s a security patch, it’s best to wait a month or two.

Predicting the Daily Closing Prices of S&P 500 ETF

After reading a recent blog post called Predicting the Daily High and low of an Exchange Traded Fund – SPY on the Business Forecasting blog written by Clive Jones, I was motivated to see how well I could forecast the same ETF ($SPY) for a month out.

Like Jones, I think forecasting the stock market is tricky business, and I have had my fair share of extremely accurate predictions, but at times, I have seen some pretty bad ones. I have tried my hand at forecasting the market for some years now, and developed many variations of forecast models to see what works best. Following in Jones’ footsteps, I would like to add to the discovery process in this seemingly crowded field and see if there is some predictability that can be found on the closing prices of SPY.

Results of the back tested forecast model:

SPYforecast

The above chart shows predictions for a 20 day period between 12/15/14 and 01/13/15. The forecast model was 98.7% accurate, and it indicates that accurate predictions can be achieved with a linear regression based on daily values.

regression

The regression stats in R were really good – especially with the R^2 value, which was .9468. An extremely high value! I believe the R^2 value was so high, because the correlation of determination table was really high, because I looked for the similar pattern matches over the last 20 years (coefficients X1-X8). The VX1-VX2 values were the VIX measures that were matched to the same time frame as the X1 and X2 values, which were the highest correlated matches to the values I was trying to predict. Then, I took the log of those values, which represents the lnVX1 and lnVX2. Lastly, the VOLX1 is the log of the volume of the best match (X1).

It is quite remarkable that there is any predictability at all, and while there is predictability, the question is can it be used as a tradable strategy? I maintain that while forecasts can be very accurate, the model should NOT be used on its own. Other technical analysis is recommended to ensure trading success.

Blizzard Warning – Getting Weather Data Using the Forecast.io API

Recently, I worked with a client to build an hourly utility forecasting system, which leverages hourly forecast data from Forecast.io. The forecast uses the hourly forecast temperatures and humidity levels for the next 48 hours. Forecast.io provides an extremely useful and rich API where users can get daily (7 days), hourly (48 hours), or even down to the minute (1 hour) weather forecasts.

The great thing about this API is users can access this data with an R package that was built by Bob Rudis (https://github.com/hrbrmstr/Rforecastio). With the blizzard hitting New York City today and tomorrow, I thought it would be a perfect time to show off the Rforecastio package in R.

As I am working on the script, I pulled the alerts for the area through the API, and it looks pretty brutal. “…CRIPPLING AND POTENTIALLY HISTORIC BLIZZARD TO IMPACT THE AREA FROM LATE TODAY INTO TUESDAY… HAZARD TYPES…HEAVY SNOW AND BLOWING SNOW…WITH BLIZZARD CONDITIONS. ACCUMULATIONS…SNOW ACCUMULATION OF 18 TO 24 INCHES…WITH LOCALLY HIGHER AMOUNTS POSSIBLE. SNOWFALL RATES OF 2 TO 4 INCHES PER HOUR EXPECTED LATE TONIGHT INTO TUESDAY MORNING.”

Let’s take a look at the temperature readings for the next 48 hours. The solid red line is the actual temp, the green line is the dew point, and the dotted red line is the wind chill.

HourlyTemps

Now, let’s take a look at the accumulation estimates for the next 48 hours. It looks there will be an average of 1-2 inches of snow falling per hour for the next 18-24 hours.

HourlyAccum

That’s a ton of snow! Hopefully, everyone can stay safe during the storm. Thoughts and prayers to the people there.

Full code: https://gist.github.com/drewgriffith15/cad4fc7ba06bd876de3c

Formatting Dates in R

For any programming language, handling date conversions can be extremely tricky if not a downright nightmare. While I will not say that R has an easy solution for all cases, the following example is meant to give an R novice a better understanding of how to easily covert date inputs into usable R date formats. Using the as.Date() function from the “base” package, dates can be imported in multiple ways: character, POSIXlt, POSIXct, and numeric formats. In most cases, if you don’t specify, R will read dates as a character. However, you could run into a situation where you need to convert a numeric date value if you were importing data from Excel.

If your dates are stored as characters, all you have to do is use the as.Date() function on the vector of dates in your dataset and supply the function with the format they are currently stored.

First example,” 09/27/99″ is in the format %m/%d/%y, while “Nov 7 1941″ is in the format %B %d %Y.

# Sample dates
dates newDates newDates # R output will be ISO standard dates in the format “%Y-%m-%d”
#[1] “1999-09-27″ “2008-03-15″

# Similar example to the last, but must be handled differently
dates newDates newDates
#[1] “1999-09-27″ “2008-03-15″

If you would have left the “%y” in lower case, the date would be wrong.

# Another example
dates newDates newDates
#[1] “1941-11-07″ “2012-04-16″

I have seen many of my clients using the numeric values from Excel, so you may have dates that are in a numeric format. You can still use as.Date to import these, which is really nice! All we have to know is the origin date that Excel starts counting from, and provide that to as.Date.

For Excel on Windows, the origin date is December 30, 1899 for dates after 1900. You might think (as would I) that you would use 1900 here, but Excel’s designer thought 1900 was a leap year, even though it really wasn’t, so they used 1899.

# from Excel
dates newDates newDates
#[1] “2013-10-20″ “2014-12-28″

Once you get the date values into R, you can change their format to the way you want to output them using the format function:

# Now, change the default output of the new dates
format(newDates, “%a %b %d %Y”)
#[1] “Sun Oct 20 2013″ “Sun Dec 28 2014″

Lastly, just for fun, I wanted to create a function that input a year in the “%Y” format, to check if this year was a leap year. Obviously, it is not, but you could check for 2016.

# Just for fun…Build function to test if input year is a leap year
is_leap_year if (year %% 400 == 0) {output = “TRUE”
}
else if (year %% 100 == 0) {output = “FALSE”
}
else if (year %% 4 == 0) {output = “TRUE”
}
else {output = “FALSE”
}
return(output)
}
# Return logical for current year
is_leap_year(as.numeric(format(Sys.time(),”%Y”)))
#[1] “FALSE”
is_leap_year(as.numeric(format(Sys.time(),”%Y”))+1)
#[1] “TRUE”

The possible date segment formats are listed in a table below:

Date Formats
Conversion specification Description Example
%a Abbreviated weekday Sun, Thu
%A Full weekday Sunday, Thursday
%b or %h Abbreviated month May, Jul
%B Full month May, July
%d Day of the month 27, 07
31-Jan
%j Day of the year 148, 188
001-366
%m Month 05, 07
12-Jan
%U Week 22, 27
Jan-53
with Sunday as first day of the week
%w Weekday 0, 4
0-6
Sunday is 0
%W Week 21, 27
00-53
with Monday as first day of the week
%x Date, locale-specific  
%y Year without century 84, 05
00-99
%Y Year with century 1984, 2005
on input:
00 to 68 prefixed by 20
69 to 99 prefixed by 19
%C Century 19, 20
%D Date formatted %m/%d/%y 05/27/84, 07/07/05
%u Weekday 7, 4
7-Jan
Monday is 1
%n Newline on output or  
Arbitrary whitespace on input
%t Tab on output or  
Arbitrary whitespace on input

 

Source code: https://gist.github.com/drewgriffith15/f85dfe3b521cc99cae3c

Estimating Stock Price Movements with Fibonacci Retracements

After reading a MarketWatch article recently, I decided to try to determine a way to calculate Fibonacci Retracements in R. If you are not familiar with Fibonacci Retracements, here is a link from StockCharts that explains what Fibonacci Retracements are, how they are calculated, and how they are used in trading. Without going into too much detail, many researchers have noticed that Fibonacci numbers can regularly observed in nature, and interestingly enough, market technicians have found that it often predicts the next move of a stock price and helps determine potential reversal levels for stocks.

I did a quick search to see if anyone had attempted building a Fibonacci Retracement script before in R, and although I found some articles related to calculating the raw Fibonacci numbers, I didn’t return any results that were related to what I was trying to do.

The calculations were much more simple than expected.

In this example, we are looking at SPY, and the stock is in a clear uptrend. We get the lowest swing, point A, 181.92; highest swing, point B, 208.97. Now, the idea is to calculate the retracements 38.2%, 50.0%, 61.8%, because stocks, more often than not, tend to dip into one of those three ranges on a pullback. That is where we would want to buy!

fibSPY

Using the chart above as an example, here are the calculations for an uptrend:

First find low and high: 208.97 — 181.92 = 27.05

0.382 (38.2%) retracement = 208.97 — 27.05 x 0.382 = 198.63

0.500 (50.0%) retracement = 208.97 — 27.05 x 0.500 = 195.45

0.618 (61.8%) retracement = 208.97 — 27.05 x 0.618 = 192.25

From the looks of it, in this current example, SPY could be ready for a bounce off of the 38% retracement level. It seems like there is good support at the 198.5 range. Trading from Friday’s session indicates that a move high from its current price is likely.

I built an R script that looks at the last 90 trading days, and automatically calculates the Fibonacci Retracements based on the highest and lowest price within that timeframe. For the full R code, see here: https://gist.github.com/drewgriffith15/e34560476a022612aa60