Attention iPhone users: iOS 15.0 through 15.2.x contain bugs that affect the auto-open feature. Versions above or below work fine.

System outage

Comments

109 comments

  • Ktwalsh52

    I use IT but have no tech education. What I do understand is that even with this outage I probably have lost more time due to Comcast outages in my area than Tailwind. I pay Comcast over $200 per month and they haven’t provided any compensation for the time they are down, so what do you expect for a one time charge of $100. Scott will get this fixed and we will be back to having the best product on the market with no monthly fee.

    How do some of you clowns deal with real life issues if you think this is a crisis?

    7
    Comment actions Permalink
  • Scott Riesebosch

    Hi Everyone,

    I have another update. We were able to fix the certificate issue in the early hours this morning. We brought everything online as a test a couple hours ago and everything came back up, and we sent some test commands through. Everything worked. We took it back down for a bit to make some additional adjustments, but we have resolved the single largest issue, which was certificate related.

    The main point is - it came back up and we saw virtually everyone's controllers come back online this morning for a short time during our test. Once our team has completed some changes to settings and run some more tests, we will bring everything back up again and hopefully stay that way. We will continue to monitor it closely throughout the day, and i'm already preparing tomorrow's scrum meeting - with a single focus on DevOps / reliability. After this horrible experience I am laser focused on this. I'm sorry but feature requests will have to be delayed for a bit. Hopefully not too long. I've brought in some outside DevOps professionals to move things along quickly, and even some of our customers have offered to help. God bless IT people. What a wonderful bunch.

    To those of you in IT, I salute you. I am a hardware guy. IT / software is a tough job, and when things go wrong thousands of people want answers - and they deserve answers. This is why I've tried to keep people informed along the way the best I can.

    We learned some things on the last outage and we were in the process of implementing some of them when this one occurred. None of us like these fire drills, so please know that I have, and will continue to put more resources into reliability. Tailwind is not run on a server in my garage contrary to what some folks have said to me. Our servers are on AWS, and we do have a backup server.

    Looking forward to putting this one to bed in the coming hours so we can get some sleep.

    We will push out announcements once things are up and running and appear to be back to normal. You may already be seeing your devices back online right now, but the team is still making some adjustments so please try to refrain from saying "Hey might light is green - I tried it and it still doesn't work". We will announce when it is stable again :)

    Thank you so much for your kindness and patience. It really does mean a lot to me, my family, and the team here behind the scenes.

    5
    Comment actions Permalink
  • Lvlucky

    FYi - thanks for the follow up and my light went green this morning - just checked the app on my phone and it's up and running too - so it looks like the problem has been resolved.  Hope folks will stop complaining and move on.  Thanks for all the hard work to get it all back to go and up and running.

    0
    Comment actions Permalink
  • Sam N

    Glad to hear you can finally get some rest! We've been through what you have and it's a huge relief coming out with a fix. Please keep us updated on the local operation mode feature request. I think this will be the #1 requested item coming out of this outage.

    0
    Comment actions Permalink
  • Derek

    Scott - I initially came here when the outage started to gripe (see my first comment). But after seeing your responses and learning a few things from others here, my perspective has changed. It’s obvious you’re dedicated to and passionate about your product. Which means I can be, too. My first instinct when I see a product or service have multiple issues is to jump to the conclusion that the company just doesn’t care about providing a good experience. That’s obviously not the case here. So I’m still very happy to be on #TeamTailwimd. Excited to see how things improve and evolve based on the lessons from this experience.

    5
    Comment actions Permalink
  • sam tian

    "Once our team has completed some changes to settings and run some more tests, we will bring everything back up again and hopefully stay that way. "

    I clicked the forgot my password button and received url link. When click the url, it shows certificate not trusted, after I trust the certificate it shows 404 page not found. Maybe you have not fixed every thing or have not brought everything back up online?
    Just thought worth to let you know.

    0
    Comment actions Permalink
  • Scott Riesebosch

    Thanks Sam. All feedback is certainly useful. Yes, not everything is back up yet. Our first step was to get devices connected through the app again. There are still other APIs / features not tested. I know the IFTTT integration is still not up yet either, but Google Home and Alexa are.

    The day isn't over for us yet. Still work to be done. I'm just so happy there are thousands and thousands of green lights again

    1
    Comment actions Permalink
  • Peter Neame

    Thanks Scott for your work - a bit of an epic, reading between the lines! We were very excited to see the green light - almost as excited as when we first saw it when installed early this year (and thanks for the great instructions). I'll look forward to seeing updates/hardware that allows local control without the internet. One of my pet phobias about the modern world is the horrendous need for devices of all flavors to be able to talk to to DNS servers/certificate authorities/who knows what. 

    Thanks again

    1
    Comment actions Permalink
  • Dave R

    Scott,

    Anyone who has worked on the operational end of IT can appreciate what you have gone through over the last few days. Your communication here has helped all of us understand what has been going on with the service. Thanks for that.

    My advice is get some sleep. Then take some time to explain to all your customers what you learned from the outage and how you plan to prevent it from happening from again.

    Some customers may feel negatively about Tailwind in the short run because of an outage like this one. However experience shows that many customers can have a positive long term view if companies communicate well in the aftermath.

    1
    Comment actions Permalink
  • Tudor Coserea

    With the service restored, it would me nice to have a postmortem explaining what went wrong (please don't spare the technical details) and what things are currently done to prevent that happening in the future. I've got this solution mainly because I have access to it free of charge but having this kind of reliability issues makes it useless for practical applications. 

    0
    Comment actions Permalink
  • Joe Aklinski

    SERVICE IS DOWN AGAIN?

     

    1
    Comment actions Permalink
  • David Cushnie

    Up briefly but down again? Getting a lot of feedback from my family. They are now saying I made a bad decision since this seems to have a single point of failure. Hope it’s back up by the morning and stay that way.

    0
    Comment actions Permalink
  • Scott B

    Light is green, and my app is currently working and showing correct door status, push notification is not doing anything apparently for the last couple of hours since I have cycled the door several times in the last 2 hours.

    0
    Comment actions Permalink
  • P

    +1 for a post mortem thread and improvements to counter it.

    Also push notifications and door cycling is still working for me

    0
    Comment actions Permalink
  • Jason Parish

    Scott Riesebosch, while I really appreciate the thought process behind your decisions I want to provide some feedback about the access dilemma.  I believe the risk is acceptable, I also believe that if I provide access to another individual it is then my responsibility to control access.  The bottom line is that if it were to become a serious concern I could always unplug the device to disallow access. Perhaps the device could be put into local access only situation where those types of users' settings could be manipulated.

    0
    Comment actions Permalink
  • Joe Aklinski

    Am I the only one who is still have outage issues? It was working well earlier today and it is now down again. I've performed a reset to no avail.

    0
    Comment actions Permalink
  • Scott B

    @ Jason, and Scott R,

    Assuming that you need to be connected to the server to create a new user, couldn't this be served by adding a couple of check boxes to enrollment for LAN and/or WAN access for the new user?
    Seems like we could use a separate thread for this discussion...

    0
    Comment actions Permalink
  • Cain Brewer

    Joe-- same issues here. Seemed okay earlier today and is now down again. Not sure what to try at this point-- reset, re-entered network details, no dice.

    0
    Comment actions Permalink
  • Jeff Walton

    Was up earlier today but down as of 9PM CST. In Minnesota.

    0
    Comment actions Permalink
  • Jeff G

    My opener has been showing as offline for the last several hours as well, at first thought it might be my wi-fi but after troubleshooting and seeing other users on here seems safe to say the servers are down again.  Not to pour on when Scott's obviously having a tough few days, but really need to work on getting those email blasts and push notifications blasts out, could have avoiding going through the troubleshooting on my end if I had known it was a server side issue.

    0
    Comment actions Permalink
  • Benjamin Lam

    I'm on the east coast of the US and the system back up for a few hours this afternoon. Now it back down, I purchased this item because it was superior with many features compared to other smart garages. But all those features have been more than a headache.

    First, sharing access has been a pain to set up since everyone needs to chose their own setting. That is overthinking in my opinion 90% of the time the share person doesn't want to set up the item themselves, they just want to be able to adjust it as they feel. The owner vehicles and distance setting should have been the default setting and if the user wants to change they then can do that instead of each user setting up their own.

    Second, the auto-close was a great idea but they forgot a fail-safe if you have 2 car garage doors, one person comes home and another person leaves. The door module doesn't recognize the sensor of the arriving car just opens the door. It only sees someone leaving and closed the door. It has almost caused an accident because while the car is pulling into the garage the door closed and thankful for the door sensor on the ground stopped the door from closing. The arriving sensor or open command should always override the closing command for a small amount of time just enough that a car can pull in. Most houses in the US now have a double car garage instead of a single car garage, so for this fail-safe not implement is really poor situation thinking.

    Third, my house to the front of my neighborhood is about 0.2 to 0.4 miles. I'm not sure how the sensor works since I have the iPhone but I believe that the sensor reacts to the phone internet/LTE signal change and built-in GPS to locate if someone arrives or leaving. Because sometimes when I drove by my neighborhood the door either open or closed. For example, if someone is at home and has the garage open, the door sometimes will close if I drove by the front of my neighborhood or open if the door closes. The auto close and open I have set it at 400 feet, I ran the diagnostic just to see the accuracy of the system and it told me about 16.4 ft.

    Fourth, the battery of the sensor. The new feature was put in a few months ago that you can adjust the range of the sensor. The sensor battery I have only lasted about a few weeks after the change. Now I have set all my sensors to short-range to keep the battery last longer. I starting to track to date to see how long it lasts since the original battery come with the sensors were only worked for 2 months from the day I got the system on August 1st. The advertisement was about 1 year with regular use, but that hasn't been the case for me. Right now the new battery I replaced has held up good after 2 months it still says about 90%.

    Finally, today I don't know if this is a new feature or the system finishing a command during the system down period, but the garage door closed by itself without a sensor for iPhone users. Twice today, I have to manually stop the door from closing by unplugging the power wire. I love the system when it works, but it is so inconsistent that I'm really close to requesting a refund. I do understand that as a user you can't depend on the system 100% of the time, but right now the product performance has not justified the extra cost. I hope this system comes back online soon and some of the new fixes will resolve these situations for me. I do believe in the company because I see the potential of the product when it working. My hope is for them to continue to improve.

    0
    Comment actions Permalink
  • Scott B

    @ Jeff Walton,
    Carver county MN here, just checked again now and it is still the same for me as it was a couple of hours ago in my last post.
    Door status and app control is fine, light is green, and push notifications still not working.
    Comcast if that makes a difference...

    0
    Comment actions Permalink
  • Michael Teator

    Down again ☹️

    0
    Comment actions Permalink
  • Wendell Phillips

    Mine went back down sometime after 1900 Central time.

    0
    Comment actions Permalink
  • Stacy Gaddis

    I get things happen.  Have been in IT for over 18 years now.  But I can only say...wow.  I love this product, but what I don't like is the unexpected outages, with LITTLE TO NO NOTIFICATION.  As discussed....emails, text messages, talk to your clients.  This outage has gone on far too long.  That is the truth. 

    1
    Comment actions Permalink
  • Joe Aklinski

    Still down in Arizona 5am. Should I start shopping for a new service?

    0
    Comment actions Permalink
  • Scott Riesebosch

    Good morning everyone. I woke up this morning to panic messages again. We restored our security certificate only to discover this morning that it also has been invalidated. That is why the devices are not connecting. We are working on the certificate issue with our certificate provider. We currently do not know why they keep invalidating it, but that is the issue we keep facing. Our server is running fine - no issues there.

    More to come.

    3
    Comment actions Permalink
  • kevin whited

    I'm a new user whose device arrived yesterday morning (sans the additional door kit I ordered, which is another issue), and it seemed to install, connect to my router, and go green/yellow light just fine. Pretty slick, actually.

    Until it was time to test in the app and I kept getting the cryptic "Seems garage door opener is offline" message. 

    I checked everything and even ran through the setup steps again and finally submitted a support ticket.

    Then I discovered this community *wave* and it seems like my issue is part of a bigger issue? Hope it's resolved soon - it's always a little disappointing when something doesn't work after self installation, but I think maybe it's safe to assume I didn't screw up anything? :)

    0
    Comment actions Permalink
  • Tony Adams

    Kevin, I think your are correct. I also setup line yesterday and have the yellow light and the app telling me the garage door is offline. I hope this issue is resolved shortly as I want to be able to play with this device 😂

    Maybe this is a good case for having a local offline api? (I'm a hubitat user and saw that this local api is on the roadmap, which is one of the reasons I purchased this over others)

    2
    Comment actions Permalink
  • Scott Riesebosch

    Hi Everyone,

    I want to keep you all informed. I do not have any further details at the moment. There is something strange going on with the SSL certificate. I've brought in 3 SSL certificate specialists. More to follow.

    1
    Comment actions Permalink

Please sign in to leave a comment.